RE: 2nd PostgreSQL server in WAL shipping cluster fails to start - Mailing list pgsql-admin

Dear Alvaro and Anton,

Thank you for your quick replies, really appreciated. I ran some more tests today and the results are as follows:

Available disk space on primary server : 101GB free in the data directory. I believe some cleaning up has been done, I
needto check with my colleagues about that. 
Wal_sender_timeout : #wal_sender_timeout = 60s
Wal_keep_segments : not present in postgresql.conf
Networking issues : none at the moment, the primary server can ping the server in standby mode by its IP address. Name
resolutiondoesn't work though. 
Monitor replication lag between the two nodes:
- Primary node :

SELECT pg_current_wal_lsn();
 pg_current_wal_lsn
--------------------
 18/B6C404B8

- Standby node : TODO when the node is up again

I'm tempted to give the pgbasebackup command a go: https://www.postgresql.org/docs/13/app-pgbasebackup.html

pg_basebackup -h primary_host -D /some_app/data -U postgres -v -P

@Alvaro Herrera : the archive_command is indeed set in the primary server, as follows:

archive_command = 'test ! -f /some_app/data/mnt/server/archivedir/%f && cp %p /some_app/data/mnt/server/archivedir/%f'

There are some files in the pg_wal directory but they only amount to 289MB. If I need to rebuild the replica from
scratch,could I use the pg_basebackup command above for that purpose ? 

And you are obviously right about the error message in French. Please find it below:

2025-01-09 09:46:35.742 CET [3147382] FATAL:  n'a pas pu recevoir des données du flux de WAL : ERREUR:  le segment
demandédu journal de transaction, 000000010000000D000000D6, a déjà été supprimé 
2025-01-09 09:46:40.745 CET [3147395] LOG:  démarré le flux des journaux depuis le principal à D/D6000000 sur la
timeline1 
2025-01-09 09:46:40.745 CET [3147395] FATAL:  n'a pas pu recevoir des données du flux de WAL : ERREUR:  le segment
demandédu journal de transaction, 000000010000000D000000D6, a déjà été supprimé 
2025-01-09 09:46:45.749 CET [3147397] LOG:  démarré le flux des journaux depuis le principal à D/D6000000 sur la
timeline1 
2025-01-09 09:46:45.749 CET [3147397] FATAL:  n'a pas pu recevoir des données du flux de WAL : ERREUR:  le segment
demandédu journal de transaction, 000000010000000D000000D6, a déjà été supprimé 
2025-01-09 09:46:50.753 CET [3147424] LOG:  démarré le flux des journaux depuis le principal à D/D6000000 sur la
timeline1 
2025-01-09 09:46:50.753 CET [3147424] FATAL:  n'a pas pu recevoir des données du flux de WAL : ERREUR:  le segment
demandédu journal de transaction, 000000010000000D000000D6, a déjà été supprimé 

Thanks again for your help.

Best regards,

Samuel

-----Message d'origine-----
De : Alvaro Herrera <alvherre@alvh.no-ip.org>
Envoyé : jeudi 9 janvier 2025 12:11
À : Samuel VISCAPI <samuel.viscapi@xelians.fr>
Cc : pgsql-admin@lists.postgresql.org
Objet : Re: 2nd PostgreSQL server in WAL shipping cluster fails to start

On 2025-Jan-09, Samuel VISCAPI wrote:

> Dear all,
>
> Some years ago I set up a two PostgreSQL 13 nodes cluster on Debian 11. This cluster uses the Write-Ahead Log
Shippingmethod. I've just been told the second server (in standby mode) is down and refuses to start again with the
followingerror message (roughly translated from French) : 
>
> 2025-01-09 09:46:35.742 CET [3147382] FATAL:  could not receive data
> from WAL stream: ERROR:  segment requested from transaction journal,
> 000000010000000D000000D6, has already been removed
> 2025-01-09 09:46:40.745 CET [3147395] LOG:  journal flow started from
> primary at D/D6000000 on timeline 1
> 2025-01-09 09:46:40.745 CET [3147395] FATAL:  could not receive data
> from WAL stream : ERROR:  segment requested from transaction journal,
> 000000010000000D000000D6, has already been removed
> 2025-01-09 09:46:45.749 CET [3147397] LOG: journal flow started from
> primary at D/D6000000 on timeline 1
> 2025-01-09 09:46:45.749 CET [3147397] FATAL:  could not receive data
> from WAL stream : ERROR:  segment requested from transaction journal,
> 000000010000000D000000D6, has already been removed
> 2025-01-09 09:46:50.753 CET [3147424] LOG:  journal flow started from
> primary at D/D6000000 on timeline 1
> 2025-01-09 09:46:50.753 CET [3147424] FATAL:  could not receive data
> from WAL stream : ERROR:  segment requested from transaction journal,
> 000000010000000D000000D6, has already been removed

This means the standby is requesting a segment that was already removed.
You may be able to find those files in a WAL archive, if you have archive_command set in the primary.  If you do, then
itwould work to copy those to the standby's pg_wal/ subdirectory.  If you don't have them, then the replica must be
rebuilt.


Note that it's not a good idea to translate the error messages when posting -- that is, it's better to post exactly
whatthe log file has. 
For helpfulness you could post a translation for the lines separately.
But the developers can find the translated messages in the source code, if they need them ... but if you translate them
yourself,there's no way to know exactly which ones they are.  Sometimes subtle differences are important cues. 

--
Álvaro Herrera               48°01'N 7°57'E  -  https://www.EnterpriseDB.com/