Thread: BUG #11949: pg_basebackup creates defective tar file
The following bug has been logged on the website: Bug reference: 11949 Logged by: Hendrik Grewe Email address: hendrik.grewe@udo.edu PostgreSQL version: 9.3.5 Operating system: Ubuntu Linux 12.04 / Kernel 3.13.0-39 Description: Hi! Somehow pg_basebackup constructs defective tar files if invoked like pg_basebackup --pgdata=/path/to/folder --format=tar --label="pg_backup on 2012.10.21" The resulting base.tar in path/to/folder is somwhat corrupted. It is possible to untar this file but it is somehow impossible to add a file to the generate base.tar file throug tar --append --file /path/to/folder/base.tar someFileToAddToArchive This is also an error within TAR, because it silently (Exit Code 0) fails to add the File `someFileToAddToArchive`. But this faulty try to add the file results in a changed/repaired base.tar. If one trys to append the file on this tampered tar it will work and the file is actually added to the tar. Bugreport to TAR project is on its way, also. Since other "handmade" tars do work on appending files, the generated base.tar file by pg_basebackup must be somhow broken. If You need additional informations feel free to contact me.
On 11/12/2014 04:36 PM, hendrik.grewe@udo.edu wrote: > The following bug has been logged on the website: > > Bug reference: 11949 > Logged by: Hendrik Grewe > Email address: hendrik.grewe@udo.edu > PostgreSQL version: 9.3.5 > Operating system: Ubuntu Linux 12.04 / Kernel 3.13.0-39 > Description: > > Hi! > Somehow pg_basebackup constructs defective tar files if invoked like > > pg_basebackup --pgdata=/path/to/folder --format=tar --label="pg_backup on > 2012.10.21" > > The resulting base.tar in path/to/folder is somwhat corrupted. > It is possible to untar this file but it is somehow impossible to add a file > to the generate base.tar file throug > > tar --append --file /path/to/folder/base.tar someFileToAddToArchive > > This is also an error within TAR, because it silently (Exit Code 0) fails to > add the File `someFileToAddToArchive`. But this faulty try to add the file > results in a changed/repaired base.tar. If one trys to append the file on > this tampered tar it will work and the file is actually added to the tar. > Bugreport to TAR project is on its way, also. https://lists.gnu.org/archive/html/bug-tar/2014-11/msg00006.html > Since other "handmade" tars do work on appending files, the generated > base.tar file by pg_basebackup must be somhow broken. The conclusion on bug-tar list seems to be that this is a bug in GNU tar, and the file created by pg_basebackup is correct. I just wanted to add that you can work-around this by passing the "-b 1" option to tar. With that, it seems to work. - Heikki
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 03.12.2014 um 11:55 schrieb Heikki Linnakangas: > > The conclusion on bug-tar list seems to be that this is a bug in > GNU tar, and the file created by pg_basebackup is correct. > > I just wanted to add that you can work-around this by passing the > "-b 1" option to tar. With that, it seems to work. So the structral defect within the tarfile: https://lists.gnu.org/archive/html/bug-tar/2014-11/msg00007.html Is not caused by pg_basebackup? -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQEcBAEBAgAGBQJUfvIpAAoJEM8IZdIKCW97szMH/j1eCusgFsT11n6ax1RlM+r2 qraL1BFAf+KY2eq6WyS0iJAbRCGcAXj7BGMPjbVKeqEkdwrHTcjicfbJZFPgtiib 9BQCmTOreXZGUaAyButl3i1NvCYYivAWa0YPYA1qwzLj9MRNctMmPMsHuYwv47eG XWK5cBXr4kCDv+GKFmUq2BXOYGWRjIJzdOv/e+efZIvP7CU5AiAxqzqp7MFJSx1H YCRtBLBURqO8fsfMVESsLBwNRMtucRCZyyNe+Uzkj+N/o3SzjgCcXIjhIYu4Fbih kdCUAVH09TEWLPn2zKDLlTI/am8l2tDi+TsY86Uo8FWuRvPXcnslLynqhPnwBJM= =+Nbk -----END PGP SIGNATURE-----
On 12/03/2014 01:21 PM, Hendrik Grewe wrote: > Am 03.12.2014 um 11:55 schrieb Heikki Linnakangas: >> >> The conclusion on bug-tar list seems to be that this is a bug in >> GNU tar, and the file created by pg_basebackup is correct. >> >> I just wanted to add that you can work-around this by passing the >> "-b 1" option to tar. With that, it seems to work. > > So the structral defect within the tarfile: > https://lists.gnu.org/archive/html/bug-tar/2014-11/msg00007.html > > Is not caused by pg_basebackup? You're referring to this, from Jörg Schilling: > The only structural defect in the tarfile is that it contains too many bits in > the mode field. That shouldn't cause the --append problem. The --append problem happens because of GNU tar's bug in handling an archive with a "blocking factor" different from its default. The "too many bits in the mode field" problem seems to be that we store the file type bits that stat(2) returns in st_mode & ST_IFMT, in the archive's mode field, while the ustar file format only expects the permission bits to be there. Tar program will ignore the extra bits, which is why we haven't noticed, but I guess we should fix that anyway, for the sake of cleanliness. But that's not the root cause of the --append corruption. - Heikki
On 12/04/2014 10:24 AM, Heikki Linnakangas wrote: > The "too many bits in the mode field" problem seems to be that we store > the file type bits that stat(2) returns in st_mode & ST_IFMT, in the > archive's mode field, while the ustar file format only expects the > permission bits to be there. Tar program will ignore the extra bits, > which is why we haven't noticed, but I guess we should fix that anyway, > for the sake of cleanliness. But that's not the root cause of the > --append corruption. Ok, pushed a patch to mask out the extraneous bits in the file mode field. I only pushed it to git master, since this hasn't caused any problems in practice. - Heikki