Thread: Missing WALs when doing pg_basebackup from slave...

Missing WALs when doing pg_basebackup from slave...

From
Date:
I have been testing a master slave configuration of PostgreSQL 9.4.2 using
streaming replciation. The master is initialised with a blank database and
the slave is created with a pg_start_backup()/rsync/pg_stop_backup()
procedure. The slave connects to the master and works flawlessly. For
testing purposes backup is configured to run every 5 minutes from the
master on one location, and from the slave on an other location. The backup
procedure is done using pg_basebackup with the --xlog-method=stream to
fetch all WALs that are needed for the backup to be fully functional.

I left it running for a day. A few backups that were made from the slave
didn't contain any WAL files (only history files in complex cases when the
master had switched to a newer timeline).

Is it normal that pg_basebackup runs successfully (rc=0) and there is no
WAL files present?

The master and slave are sitting idle, after only a few transaction on the
master at the beginning of the day. I noted that all WAL switches are
caused by the backup running on the master. Is it possible the slave is in
a consistent state when it has applied all changes from the previous WAL
and the new WAL hasn't been created yet on the master (so actually no WAL-s
are needed to restore it to a consistent state)?

Regards,
Mladen Marinović


Re: Missing WALs when doing pg_basebackup from slave...

From
Venkata Balaji N
Date:
On Wed, Jun 10, 2015 at 6:47 PM, <marin@kset.org> wrote:

Is it normal that pg_basebackup runs successfully (rc=0) and there is no
WAL files present?

Yes, it is normal. "pg_basebackup" ensures that required WALs are backed along with the data directory. This is to ensure backup is consistent.
 
The master and slave are sitting idle, after only a few transaction on the
master at the beginning of the day. I noted that all WAL switches are
caused by the backup running on the master. Is it possible the slave is in
a consistent state when it has applied all changes from the previous WAL
and the new WAL hasn't been created yet on the master (so actually no WAL-s
are needed to restore it to a consistent state)?
 
I am not sure if I got your question correct. The amount of transactions in the master database may be low or high, the WALs will be replicated to slave.
To ensure slave is receiving all the WALs, you need to check the sync status between master and slave. If there is no new WAL generated at master, then slave must be in consistent state should have applied all the previous WALs, if not, then all the previous WALs are needed to get the slave to a consistent state. Nothing can be advised straight without knowing your replication configuration/architecture details.

Regards,
Venkata Balaji N

Fujitsu Australia



Re: Missing WALs when doing pg_basebackup from slave...

From
Date:
On Thu, 11 Jun 2015 14:48:44 +1000, Venkata Balaji N <nag1010@gmail.com>
wrote:
> On Wed, Jun 10, 2015 at 6:47 PM,  wrote:
>  Is it normal that pg_basebackup runs successfully (rc=0) and there is
no
>  WAL files present?
>
> Yes, it is normal. "pg_basebackup" ensures that required WALs are backed
> along with the data directory. This is to ensure backup is consistent.

The backup pg_basebackup made had NO WAL files. I tried to restored but it
failed without at lest the WAL file mentioned in the backup_label file. It
is my understanding that pg_basebackup --xlog-method=stream should stream
the needed WAL files, and it usually does. But as I mentioned, I had it run
all day every 5 minutes and 3/100 backups were without WAL files. I am not
sure if this is a glitch of pg_basebackup or what, but I expected at least
an error result code (rc!=0).

>   The master and slave are sitting idle, after only a few transaction on
> the
>  master at the beginning of the day. I noted that all WAL switches are
>  caused by the backup running on the master. Is it possible the slave is
> in
>  a consistent state when it has applied all changes from the previous
WAL
>  and the new WAL hasn't been created yet on the master (so actually no
> WAL-s
>  are needed to restore it to a consistent state)?
>  I am not sure if I got your question correct. The amount of
> transactions in the master database may be low or high, the WALs will be
> replicated to slave.To ensure slave is receiving all the WALs, you need
> to check the sync status between master and slave. If there is no new
WAL
> generated at master, then slave must be in consistent state should have
> applied all the previous WALs, if not, then all the previous WALs are
> needed to get the slave to a consistent state. Nothing can be advised
> straight without knowing your replication configuration/architecture
> details.
> Regards,Venkata Balaji N
> Fujitsu Australia
>
>
>
> Links:
> ------
> [1] mailto:marin@kset.org