Re: pg_basebackup stream xlog to tar - Mailing list pgsql-hackers

From Magnus Hagander
Subject Re: pg_basebackup stream xlog to tar
Date
Msg-id CABUevEyS0xkpBfhn3=KxC8eXQ-xNGdHM=_=YTp_wziMXRFy=vA@mail.gmail.com
Whole thread Raw
In response to Re: pg_basebackup stream xlog to tar  (Michael Paquier <michael.paquier@gmail.com>)
Responses Re: pg_basebackup stream xlog to tar  (Michael Paquier <michael.paquier@gmail.com>)
List pgsql-hackers


On Thu, Sep 1, 2016 at 2:39 PM, Michael Paquier <michael.paquier@gmail.com> wrote:
On Thu, Sep 1, 2016 at 5:13 PM, Magnus Hagander <magnus@hagander.net> wrote:
> We don't seem to check for similar issues as the one just found in the
> existing tests though, do we? As in, we don't actually verify that the xlog
> files being streamed are 16Mb? (Or for that matter that the tarfile emitted
> by -Ft is actually a tarfile?) Or am I missing some magic somewhere? :)

No. There is no checks on the WAL file size (you should use the output
of pg_controldata to see how large the segments should be). For the
tar file, the complication is in its untar... Perl provides some ways
to untar things, though the oldest version that we support in the TAP
tests does not offer that :(

Ugh. That would be nice to have, but I think that's outside the scope of this patch. 

PFA is an updated version of this patch that:
* documents a magic value passed to zlib (which is in their documentation as being a magic value, but has no define)
* fixes the padding of tar files
* adds a most basic test that the -X stream -Ft does produce a tarfile

As for using XLOGDIR to drive the name of the tarfile. pg_basebackup is already hardcoded to use pg_xlog. And so are the tests. We probably want to fix that, but that's a separate step and this patch will be easier to review and test if we keep it out for now.

--
Attachment

pgsql-hackers by date:

Previous
From: Michael Paquier
Date:
Subject: Re: pg_basebackup, pg_receivexlog and data durability (was: silent data loss with ext4 / all current versions)
Next
From: Simon Riggs
Date:
Subject: Re: pg_hba_file_settings view patch