Thread: pg_basebackup: could not get write-ahead log end position fromserver: ERROR

Dear Team,

1)Whilst taking full backup via pg_basebackup I have facing below error. Backup is schedule via cronjob.

================================================================================

****** Full Backup of posgresql database started Fri Jun 8 20:40:36 PKT 2018.

080618204036

pg_basebackup: initiating base backup, waiting for checkpoint to complete

pg_basebackup: checkpoint completed

pg_basebackup: write-ahead log start point: 43/400252F8 on timeline 1

17624153/17624153 kB (100%), 2/2 tablespaces                                        

pg_basebackup: could not get write-ahead log end position from server: ERROR:  requested WAL segment 000000010000004300000040 has already been removed

pg_basebackup: removing data directory "/db01/pgdata/hot_backup/080618204036"

2)Also after complete backup I have to manually delete the wal archive files using pg_archivecleanup. Is there any way to automatically delete after full backup is complete and wal archive are no longer required.

pg_archivecleanup -d /db01/wal_archive 00000001000000390000000A

Kindly let me know how to rectify.

pg_basebackup -z -X fetch --progress -v --format=t -D /db01/pgdata/hot_backup/$DD --no-password

Current Version:

                                                 version                                                

---------------------------------------------------------------------------------------------------------

PostgreSQL 10.1 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18), 64-bit

LINUX : .10.0-327.el7.x86_64

Regards,

Saeed Ahmed

Hi Nawaz,

Many thanks for the update.

1) have you configured the archive_command to a script that archives the online WAL to the archive destination and deletes the file?

Yes

2) As far as I know the cleanup of archive destination has to done manually. 

Ok

Regards,

Saeed Ahmed

Disclaimer

The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.

Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.

If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe@fast.au.fujitsu.com

That explains it, the script deletes the WAL file as soon as it is archived but pg_basebackup is not aware of this so it generates that error.

Hi Nawaz,

Many thanks for the update.

1) have you configured the archive_command to a script that archives the online WAL to the archive destination and deletes the file?

Yes

2) As far as I know the cleanup of archive destination has to done manually. 

Ok

Regards,

Saeed Ahmed

Disclaimer

The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.

Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.

If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe@fast.au.fujitsu.com

Hi,

Thanks for the support and your time.

The issue was resolved after increasing wal_keep_segments to 20 means it will keep 20 files in directory /PGDATA/pg_wal .

and ran "pg_ctl reload" to enable the configuration without rebooting the db.

Regards,

Saeed Ahmed

That explains it, the script deletes the WAL file as soon as it is archived but pg_basebackup is not aware of this so it generates that error.

Hi Nawaz,

Many thanks for the update.

1) have you configured the archive_command to a script that archives the online WAL to the archive destination and deletes the file?

Yes

2) As far as I know the cleanup of archive destination has to done manually. 

Ok

Regards,

Saeed Ahmed

Disclaimer

The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.

Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.

If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe@fast.au.fujitsu.com

Disclaimer

The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.

Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.

If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe@fast.au.fujitsu.com