pg_basebackup stream xlog to tar - Mailing list pgsql-hackers

From Magnus Hagander
Subject pg_basebackup stream xlog to tar
Date
Msg-id CABUevEx-QvhVbb+oNv_ai-K2334OPDncpqNzs0G2JyaBu1zA=Q@mail.gmail.com
Whole thread Raw
Responses Re: pg_basebackup stream xlog to tar  (Michael Paquier <michael.paquier@gmail.com>)
List pgsql-hackers
Attached patch adds support for -X stream to work with .tar and .tar.gz file formats.

If tar mode is specified, a separate pg_xlog.tar (or .tar.gz) file is created and the data is streamed into it. Regular mode is (should not) see any changes in how it works.

The implementation creates a "walmethod" for directory and one for tar, which is basically a set of function pointers that we pass around as part of the StreamCtl structure. All calls to modify the files are sent through the current method, using the normal open/read/write calls as it is now for directories, and the more complicated method for tar and targz.

The tar method doesn't support all things that are required for pg_receivexlog, but I don't think it makes any real sense to have support for pg_receivexlog in tar mode. But it does support all the things that pg_basebackup needs.

Some smaller pieces of functionality like unlinking files on failure and padding files have been moved into the walmethod because they have to be differently implemented (we cannot pre-pad a compressed file -- the size will depend on the compression ration anyway -- for example).

AFAICT we never actually documented that -X stream doesn't work with tar in the manpage of current versions. Only in the error message. We might want to fix that in backbranches.

In passing this also fixes an XXX comment about not re-lseeking on the WAL file all the time -- the walmethod now tracks the current position in the file in a variable.

Finally, to make this work, the pring_tar_number() function is now exported from port/tar.c along with the other ones already exported from there.

--
Attachment

pgsql-hackers by date:

Previous
From: Andres Freund
Date:
Subject: Re: [Patch] Temporary tables that do not bloat pg_catalog (a.k.a fast temp tables)
Next
From: Tomas Vondra
Date:
Subject: Re: [Patch] Temporary tables that do not bloat pg_catalog (a.k.a fast temp tables)