Thread: pg_start_backup: file has vanished from pg_subtrans/
Hi, While backing up some postgresql-8.4.2 instances using pg_start_backup()/pg_stop_backup() We noticed that sometimes rsync complains: file has vanished: "............./data/pg_subtrans/3A37" rsync warning: some files vanished before they could be transferred (code 24) at main.c(1039) [sender=3.0.6 We already have excluded pgsql_tmp/ pg_xlog/ and pg_stat_tmp/ so I was tempted to exclude pg_subtrans/ as well. However some googling seems to suggest that this might invalidate the backup: https://groups.google.com/forum/#!msg/pgbarman/zQxg6pOXkZ8/AmTmVtUA3dEJ I am not sure what to believe, either - my backup was invalid and I have to take measures to prevent this (and file a bug report) - I can ignore this *and* can safely exclude pg_trans/ from my backups. - there is some other workaround... -- Leo Baltus
Leo Baltus wrote: > Hi, > > While backing up some postgresql-8.4.2 instances using > pg_start_backup()/pg_stop_backup() 8.4.2 was released in 2009, so you're missing all the bugfixes till July 2014 which is when the 8.4 branch was dropped out of support -- which means you're also missing bugfixes from that point onwards. I think you need to jump to 8.4.22 (the last one in that release) because this might be a fixed bug; and you should consider moving to one of the supported branches very soon, at least 9.0 but that one's going to lose support soon too so I may suggest something even newer. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 08/07/2015 07:20 AM, Alvaro Herrera wrote: > Leo Baltus wrote: >> Hi, >> >> While backing up some postgresql-8.4.2 instances using >> pg_start_backup()/pg_stop_backup() > > 8.4.2 was released in 2009, so you're missing all the bugfixes till July > 2014 which is when the 8.4 branch was dropped out of support -- which > means you're also missing bugfixes from that point onwards. I think you > need to jump to 8.4.22 (the last one in that release) because this might > be a fixed bug; and you should consider moving to one of the supported > branches very soon, at least 9.0 but that one's going to lose support > soon too so I may suggest something even newer. That being said, I thought: file has vanished: "............./data/pg_subtrans/3A37" would not be a bug but normal behavior. -- Adrian Klaver adrian.klaver@aklaver.com
Adrian Klaver wrote: > On 08/07/2015 07:20 AM, Alvaro Herrera wrote: > >Leo Baltus wrote: > >>Hi, > >> > >>While backing up some postgresql-8.4.2 instances using > >>pg_start_backup()/pg_stop_backup() > > > >8.4.2 was released in 2009, so you're missing all the bugfixes till July > >2014 which is when the 8.4 branch was dropped out of support -- which > >means you're also missing bugfixes from that point onwards. I think you > >need to jump to 8.4.22 (the last one in that release) because this might > >be a fixed bug; and you should consider moving to one of the supported > >branches very soon, at least 9.0 but that one's going to lose support > >soon too so I may suggest something even newer. > > That being said, I thought: > > file has vanished: "............./data/pg_subtrans/3A37" > > would not be a bug but normal behavior. Well, it's true that pg_subtrans files vanish during regular running. WAL replay of the resulting backup *should* work fine regardless, and this is just rsync being paranoid, but I'm not sure that it *actually* works. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 8/7/15 7:01 AM, Leo Baltus wrote: > While backing up some postgresql-8.4.2 instances using > pg_start_backup()/pg_stop_backup() > > We noticed that sometimes rsync complains: > > file has vanished: "............./data/pg_subtrans/3A37" > rsync warning: some files vanished before they could be transferred (code 24) at main.c(1039) [sender=3.0.6 It's perfectly normal for Postgres to remove files during a backup. If a table is dropped then the files storing that table will be deleted. Files can also be removed from pg_subtrans once they are no longer needed. > We already have excluded pgsql_tmp/ pg_xlog/ and pg_stat_tmp/ so I was > tempted to exclude pg_subtrans/ as well. However some googling seems to > suggest that this might invalidate the backup: I would recommend only excluding pg_xlog as recommended in the Postgres docs. > https://groups.google.com/forum/#!msg/pgbarman/zQxg6pOXkZ8/AmTmVtUA3dEJ > > I am not sure what to believe, either > - my backup was invalid and I have to take measures to prevent this > (and file a bug report) Your backup should be good so long as you've captured all the WAL. > - I can ignore this *and* can safely exclude pg_trans/ from my backups. > - there is some other workaround... Do *not* exclude pg_subtrans. You may also want to look at backup software such as pgBackRest. It understands that files can be removed during backup and does not emit warnings when it happens (though info messages are still logged depending on your log level). -- -David david@pgmasters.net
Op 08/08/2015 om 14:24:00 -0400, schreef David Steele: > On 8/7/15 7:01 AM, Leo Baltus wrote: > >While backing up some postgresql-8.4.2 instances using > >pg_start_backup()/pg_stop_backup() > > > >We noticed that sometimes rsync complains: > > > >file has vanished: "............./data/pg_subtrans/3A37" > >rsync warning: some files vanished before they could be transferred (code 24) at main.c(1039) [sender=3.0.6 > > It's perfectly normal for Postgres to remove files during a backup. If a > table is dropped then the files storing that table will be deleted. Files > can also be removed from pg_subtrans once they are no longer needed. > > >We already have excluded pgsql_tmp/ pg_xlog/ and pg_stat_tmp/ so I was > >tempted to exclude pg_subtrans/ as well. However some googling seems to > >suggest that this might invalidate the backup: > > I would recommend only excluding pg_xlog as recommended in the Postgres > docs. > Check. > >https://groups.google.com/forum/#!msg/pgbarman/zQxg6pOXkZ8/AmTmVtUA3dEJ > > > >I am not sure what to believe, either > > - my backup was invalid and I have to take measures to prevent this > > (and file a bug report) > > Your backup should be good so long as you've captured all the WAL. > > > - I can ignore this *and* can safely exclude pg_trans/ from my backups. > > - there is some other workaround... > > Do *not* exclude pg_subtrans. Ok, for now I will ignore rsync's exit code 24. > > You may also want to look at backup software such as pgBackRest. It > understands that files can be removed during backup and does not emit > warnings when it happens (though info messages are still logged depending on > your log level). Thanks! -- Leo Baltus