RE: Postgresql error : PANIC: could not locate a valid checkpoint record - Mailing list pgsql-general

From Mahendrakar, Prabhakar - Dell Team
Subject RE: Postgresql error : PANIC: could not locate a valid checkpoint record
Date
Msg-id BYAPR19MB2886157EE36EAA32F879868594B29@BYAPR19MB2886.namprd19.prod.outlook.com
Whole thread Raw
In response to Re: Postgresql error : PANIC: could not locate a valid checkpoint record  (Mateusz Henicz <mateuszhenicz@gmail.com>)
Responses RE: Postgresql error : PANIC: could not locate a valid checkpoint record
List pgsql-general

Hello there,

 

Please find my response as below:

 

  • Also, are you able to reproduce the issue?

No we are not able to reproduce at our end. But this is happening occasionally at customer environment that has large DB size.

  • I.e. try restoring the database to state before upgrade, try to upgrade again. Does this corruption happen again?  

              Could you please let us know the process to perform this operations. Can I get some document to do the same.

 

Thanks,

Prabhakar

 

From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Tuesday, June 21, 2022 5:59 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record

 

[EXTERNAL EMAIL]

Could you also answer on questions asked by Laurenz Albe a few emails back? That could shed some light into your issue. We would know if the upgrade was performed properly and possibly point where the issue can be.

Also, are you able to reproduce the issue? I.e. try restoring the database to state before upgrade, try to upgrade again. Does this corruption happen again?  
About pg_resetwal, you are right that it should be done as a last resort. On the other hand, if your production database does not start after upgrade and you do not have any way to rollback your changes it might be it.

 

Cheers,

Mateusz

 

wt., 21 cze 2022 o 14:02 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):

Could you please provide some insights in the below query. We are in need of this as it this issue is seen at some of our customer’s site.

 

Thanks,

Prabhakar

 

 

Internal Use - Confidential

 

Internal Use - Confidential

From: Mahendrakar, Prabhakar - Dell Team
Sent: Monday, June 20, 2022 4:22 PM
To: Mateusz Henicz
Cc: pgsql-general@lists.postgresql.org
Subject: RE: Postgresql error : PANIC: could not locate a valid checkpoint record

 

Thanks for the response.

Yes, we have taken care of proper shut down of Postgres before initiating the Upgrade.

pg_resetwal – I have read that using pg_resetwal may cause the Database more inconsistent and should be used only as a last resort.

 

Also this problem ( checkpoint related issue -could not locate a valid checkpoint record ) is not happening frequently. This issue is seen with the large size of Data base.

Please let me know if you require any more information.

 

Thanks,

Prabhakar

 

From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Friday, June 17, 2022 3:39 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record

 

[EXTERNAL EMAIL]

Assuming you have shut down your postgres properly before upgrading, it should be safe for you to run pg_resetwal.

It should help in this case.

 

pt., 17 cze 2022 o 12:03 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):

Yes, We are using the pg_upgrade utility of Postgres.

 

From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Friday, June 17, 2022 3:31 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record

 

[EXTERNAL EMAIL]

Hi,

Have you done pg_upgrade post Postgres 13 installation?

 

Cheers,
Mateusz

 

pt., 17 cze 2022 o 11:20 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):

Hello,

 

Good Morning !

 

We are facing checkpoint related issues from PostGreSQL 13.4 ( could not locate a valid checkpoint record) and Postgres service fails to come up.

========================================================================================================

LOG:  starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit

LOG:  listening on IPv4 address "127.0.0.1", port 9003

LOG:  listening on IPv4 address "10.xx.xx.x..", port 9003

LOG:  listening on Unix socket "/tmp/.s.PGSQL.9003"

LOG:  database system was shut down at 2022-06-09 10:19:24 CEST

LOG:  invalid primary checkpoint record

PANIC:  could not locate a valid checkpoint record

LOG:  startup process (PID 8773) was terminated by signal 6: Aborted

LOG:  aborting startup due to startup process failure

LOG:  database system is shut down

 

This issue is seen in both Windows and Linux OS platforms.

 

To Brief on the Scenario: Our product say example with Version A1 uses Postgres 10 and in the latest version of our product (Say A2) we upgraded the Postgres to 13.

                                           When we try to upgrade our Product through InstallAnyWhere from A1 to A2, Postgres service fails with above mentioned error.

 

 Could you please suggest the probable cause of the issue. Let us know if you require any more information.

 

Thanks,

Prabhakar

 

 

Internal Use - Confidential

 

Internal Use - Confidential

 

Internal Use - Confidential

pgsql-general by date:

Previous
From: "Mahendrakar, Prabhakar - Dell Team"
Date:
Subject: RE: Postgresql error : PANIC: could not locate a valid checkpoint record
Next
From: Zb B
Date:
Subject: PostgreSQL with Patroni not replicating to all nodes after adding 3rd node (another secondary)