PG version 16.14 upgrade went recovery state - Mailing list pgsql-admin

From K P Manoj
Subject PG version 16.14 upgrade went recovery state
Date
Msg-id CAMVgnQ48N4Vspjg-m4TgYEK2-O_F5L0gGEPtjVSMqQrUQ+24Dg@mail.gmail.com
Whole thread
Responses Re: PG version 16.14 upgrade went recovery state
List pgsql-admin

K P Manoj kpmanojpg@gmail.com

12:24 PM (3 hours ago)
to pgsql-hackerspgsql-bugs
Hi Team,

I am encountering an issue when upgrading our PostgreSQL database from minor version 16.10 to 16.14.

During the upgrade, the database enters an indefinite recovery mode across the master, standby, and multiple cascading replicas. Due to legacy requirements, we are utilizing streaming replication for this setup. Interestingly, when performing the same migration to version 16.13, the process completes successfully without any issues.

Could you please confirm if there are any known bugs regarding this behavior in version 16.14?

For reference, please find the details of our test steps and logs below.

1. Upgrade Attempt to Version 16.14 (Database Stays in Recovery Mode)

Version Check:

root@pgreplica-master-0:/var/lib/postgresql/data/pg_wal# /usr/local/pgsql/bin/psql --version
psql (PostgreSQL) 16.14

Process Status:

root@pgreplica-master-0:/var/lib/postgresql/data/pg_wal# ps -ef|grep rec
postgres      48      43  1 15:28 ?        00:00:01 postgres: startup recovering 0000000100002499000000E4
root         166      58  0 15:30 pts/0    00:00:00 grep --color=auto rec

WAL Directory Details:

root@pgreplica-master-0:/var/lib/postgresql/data/pg_wal# ls -l 0000000100002499000000E4
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:25 0000000100002499000000E4

root@pgreplica-master-0:/var/lib/postgresql/data/pg_wal# ls -lt|more
total 55001312
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:54 000000010000249A00000059
drwxrws--- 2 postgres postgres   225280 Jul  2 14:54 archive_status
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:54 000000010000249A00000058
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:54 000000010000249A00000057
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:54 000000010000249A00000056
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:54 000000010000249A00000055
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A00000054
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A00000053
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A00000052
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A00000051
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A00000050
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A0000004F
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A0000004E
-rw-rw---- 1 postgres postgres 16777216 Jul  2 14:53 000000010000249A0000004D

PostgreSQL Logs:

2026-07-02 15:28:38.709 UTC [43]: [3-1] user=,db=  >LOG:  starting PostgreSQL 16.14 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
2026-07-02 15:28:38.709 UTC [43]: [4-1] user=,db=  >LOG:  listening on IPv4 address "0.0.0.0", port 5432
2026-07-02 15:28:38.709 UTC [43]: [5-1] user=,db=  >LOG:  listening on IPv6 address "::", port 5432
2026-07-02 15:28:38.722 UTC [43]: [6-1] user=,db=  >LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2026-07-02 15:28:38.753 UTC [48]: [1-1] user=,db=  >LOG:  database system was interrupted while in recovery at log time 2026-07-02 14:22:09 UTC
2026-07-02 15:28:38.753 UTC [48]: [2-1] user=,db=  >HINT:  If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.
2026-07-02 15:28:38.979 UTC [48]: [3-1] user=,db=  >LOG:  entering standby mode
2026-07-02 15:28:38.993 UTC [48]: [4-1] user=,db=  >LOG:  redo starts at 2499/CFF32450
2026-07-02 15:28:47.282 UTC [49]: [1-1] user=127.0.0.1,db=postgres  >FATAL:  the database system is not yet accepting connections
2026-07-02 15:28:47.282 UTC [49]: [2-1] user=127.0.0.1,db=postgres  >DETAIL:  Consistent recovery state has not been yet reached.
2026-07-02 15:28:57.291 UTC [57]: [1-1] user=127.0.0.1,db=postgres  >FATAL:  the database system is not yet accepting connections
2026-07-02 15:28:57.291 UTC [57]: [2-1] user=127.0.0.1,db=postgres  >DETAIL:  Consistent recovery state has not been yet reached.

2. Successful Migration to Version 16.13 (Streaming Replication Working)

Version Check:

root@pgreplica-master-0:/# psql --version
psql (PostgreSQL) 16.13 (Debian 16.13-1.pgdg12+1)

Process Status (Walreceiver Streaming Active):

root@pgreplica-master-0:/var/lib/postgresql/data/pg_wal# ps -ef|grep rec
postgres      48      43 32 15:40 ?        00:00:20 postgres: startup recovering 000000010000249B0000002E
postgres      50      43  7 15:40 ?        00:00:03 postgres: walreceiver streaming 249B/2E660000
root         105      80  0 15:41 pts/0    00:00:00 grep rec

WAL File Check:

root@pgreplica-master-0:/var/lib/postgresql/data/pg_wal# ls -lth 000000010000249B0000002E
-rw------- 1 postgres postgres 16M Jul  2 15:41 000000010000249B0000002E

Database Connection Verification:

root@pgreplica-master-0:/var/lib/postgresql/data/pg_wal# psql -U postgres -c " select version () "
                                                           version                                                              
------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 16.13 (Debian 16.13-1.pgdg12+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14+deb12u1) 12.2.0, 64-bit
(1 row)

Please let me know if any further details or logs are required to investigate this issue.

Best regards,

Manoj K P

pgsql-admin by date:

Previous
From: Raj
Date:
Subject: Major upgrade in patroni cluster
Next
From: Laurenz Albe
Date:
Subject: Re: PG version 16.14 upgrade went recovery state