Thread: tar error while running basebackup

Hi!

We're using two backup strategies to get consistent backups of our
postgresql databases. First, we create a complete dump every night by
running pg_dump, zipping the file and writing this backup on tape.
Second, we create a basebackup every saturday. To create a basebackup,
we run pg_start_backup. After that, we create a tar file of the complete
database directory and stop the backup mode by running pg_stop_backup.
Of course, all archived wal logs are also copied ;)

Well, everything was fine for month. But from time to time, I get an
error when running tar:

tar: ./base/208106/209062: File shrank by 262144 bytes; padding with zeros
tar: ./base/208106/210576: file changed as we read it
tar: ./base/208106/210577: file changed as we read it
tar: ./base/208106/210431: file changed as we read it

How can this happen? I always thought, that, when in backup mode,
nothing is able to change the database - so the database files shouldn't
change. Can autovaccumdb cause the changes?
I already read something, that this kind of errors can be ignored when
creating a basebackup, but I can't believe that. Normally, the tar file
have to be worthless, when an error occurs - or do I have an error in
reasoning?

best regards,
Andre

> From: Andre Brandt <brandt@decoit.de>
> Subject: [GENERAL] tar error while running basebackup
> To: pgsql-general@postgresql.org
> Date: Tuesday, 13 October, 2009, 11:40 AM
> Hi!
>
> We're using two backup strategies to get consistent backups
> of our postgresql databases. First, we create a complete
> dump every night by running pg_dump, zipping the file and
> writing this backup on tape.
> Second, we create a basebackup every saturday. To create a
> basebackup, we run pg_start_backup. After that, we create a
> tar file of the complete database directory and stop the
> backup mode by running pg_stop_backup. Of course, all
> archived wal logs are also copied ;)
>
> Well, everything was fine for month. But from time to time,
> I get an error when running tar:
>
> tar: ./base/208106/209062: File shrank by 262144 bytes;
> padding with zeros
> tar: ./base/208106/210576: file changed as we read it
> tar: ./base/208106/210577: file changed as we read it
> tar: ./base/208106/210431: file changed as we read it
>
> How can this happen? I always thought, that, when in backup
> mode, nothing is able to change the database - so the
> database files shouldn't change. Can autovaccumdb cause the
> changes?
> I already read something, that this kind of errors can be
> ignored when creating a basebackup, but I can't believe
> that. Normally, the tar file have to be worthless, when an
> error occurs - or do I have an error in reasoning?

My understanding was that when you back up like this you are actually taring up inconsistant database files, but it
doesn'tmatter as you have the wal logs to replay any changes, thus correcting the inconsistancies. 

On Tue, Oct 13, 2009 at 12:40:37PM +0200, Andre Brandt wrote:
> tar: ./base/208106/209062: File shrank by 262144 bytes; padding with zeros
> tar: ./base/208106/210576: file changed as we read it
> tar: ./base/208106/210577: file changed as we read it
> tar: ./base/208106/210431: file changed as we read it
>

This is entirely normal.

> How can this happen? I always thought, that, when in backup mode,
> nothing is able to change the database - so the database files shouldn't
> change. Can autovaccumdb cause the changes?

pg_start_backup() doesn't tell the database to stop writing changes to disk;
it essentially just says "perform a checkpoint", which means all changes as of
that instant are written to the base data files. That ensures that you start
your base backup in a consistent state. When you recover it, replaying the WAL
files will fix any weirdness in your base backup, and you'll get a working
database, current up to the last WAL file you recovered.

--
Joshua Tolley / eggyknap
End Point Corporation
http://www.endpoint.com

>> How can this happen? I always thought, that, when in backup mode,
>> nothing is able to change the database - so the database files shouldn't
>> change. Can autovaccumdb cause the changes?
>>
>
> pg_start_backup() doesn't tell the database to stop writing changes to disk;
> it essentially just says "perform a checkpoint", which means all changes as of
> that instant are written to the base data files. That ensures that you start
> your base backup in a consistent state. When you recover it, replaying the WAL
> files will fix any weirdness in your base backup, and you'll get a working
> database, current up to the last WAL file you recovered.
>
>

I think, I will install a newer version of tar on my cluster. It seems,
that redhat shipped a really old version (1.15.1) with RHEL5. In version
1.16, the return code was changed when an read problem occurs - so I can
differ between an read error and an fatal error.

Thx for your fast help =)