Thread: Postgresql error : PANIC: could not locate a valid checkpoint record
Postgresql error : PANIC: could not locate a valid checkpoint record
Hello,
Good Morning !
We are facing checkpoint related issues from PostGreSQL 13.4 ( could not locate a valid checkpoint record) and Postgres service fails to come up.
========================================================================================================
LOG: starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit
LOG: listening on IPv4 address "127.0.0.1", port 9003
LOG: listening on IPv4 address "10.xx.xx.x..", port 9003
LOG: listening on Unix socket "/tmp/.s.PGSQL.9003"
LOG: database system was shut down at 2022-06-09 10:19:24 CEST
LOG: invalid primary checkpoint record
PANIC: could not locate a valid checkpoint record
LOG: startup process (PID 8773) was terminated by signal 6: Aborted
LOG: aborting startup due to startup process failure
LOG: database system is shut down
This issue is seen in both Windows and Linux OS platforms.
To Brief on the Scenario: Our product say example with Version A1 uses Postgres 10 and in the latest version of our product (Say A2) we upgraded the Postgres to 13.
When we try to upgrade our Product through InstallAnyWhere from A1 to A2, Postgres service fails with above mentioned error.
Could you please suggest the probable cause of the issue. Let us know if you require any more information.
Thanks,
Prabhakar
Internal Use - Confidential
Mateusz
Hello,
Good Morning !
We are facing checkpoint related issues from PostGreSQL 13.4 ( could not locate a valid checkpoint record) and Postgres service fails to come up.
========================================================================================================
LOG: starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit
LOG: listening on IPv4 address "127.0.0.1", port 9003
LOG: listening on IPv4 address "10.xx.xx.x..", port 9003
LOG: listening on Unix socket "/tmp/.s.PGSQL.9003"
LOG: database system was shut down at 2022-06-09 10:19:24 CEST
LOG: invalid primary checkpoint record
PANIC: could not locate a valid checkpoint record
LOG: startup process (PID 8773) was terminated by signal 6: Aborted
LOG: aborting startup due to startup process failure
LOG: database system is shut down
This issue is seen in both Windows and Linux OS platforms.
To Brief on the Scenario: Our product say example with Version A1 uses Postgres 10 and in the latest version of our product (Say A2) we upgraded the Postgres to 13.
When we try to upgrade our Product through InstallAnyWhere from A1 to A2, Postgres service fails with above mentioned error.
Could you please suggest the probable cause of the issue. Let us know if you require any more information.
Thanks,
Prabhakar
Internal Use - Confidential
RE: Postgresql error : PANIC: could not locate a valid checkpoint record
Yes, We are using the pg_upgrade utility of Postgres.
From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Friday, June 17, 2022 3:31 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record
[EXTERNAL EMAIL]
Hi,
Have you done pg_upgrade post Postgres 13 installation?
Cheers,
Mateusz
pt., 17 cze 2022 o 11:20 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):
Hello,
Good Morning !
We are facing checkpoint related issues from PostGreSQL 13.4 ( could not locate a valid checkpoint record) and Postgres service fails to come up.
========================================================================================================
LOG: starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit
LOG: listening on IPv4 address "127.0.0.1", port 9003
LOG: listening on IPv4 address "10.xx.xx.x..", port 9003
LOG: listening on Unix socket "/tmp/.s.PGSQL.9003"
LOG: database system was shut down at 2022-06-09 10:19:24 CEST
LOG: invalid primary checkpoint record
PANIC: could not locate a valid checkpoint record
LOG: startup process (PID 8773) was terminated by signal 6: Aborted
LOG: aborting startup due to startup process failure
LOG: database system is shut down
This issue is seen in both Windows and Linux OS platforms.
To Brief on the Scenario: Our product say example with Version A1 uses Postgres 10 and in the latest version of our product (Say A2) we upgraded the Postgres to 13.
When we try to upgrade our Product through InstallAnyWhere from A1 to A2, Postgres service fails with above mentioned error.
Could you please suggest the probable cause of the issue. Let us know if you require any more information.
Thanks,
Prabhakar
Internal Use - Confidential
Internal Use - Confidential
Yes, We are using the pg_upgrade utility of Postgres.
From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Friday, June 17, 2022 3:31 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record
[EXTERNAL EMAIL]
Hi,
Have you done pg_upgrade post Postgres 13 installation?
Cheers,
Mateusz
pt., 17 cze 2022 o 11:20 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):
Hello,
Good Morning !
We are facing checkpoint related issues from PostGreSQL 13.4 ( could not locate a valid checkpoint record) and Postgres service fails to come up.
========================================================================================================
LOG: starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit
LOG: listening on IPv4 address "127.0.0.1", port 9003
LOG: listening on IPv4 address "10.xx.xx.x..", port 9003
LOG: listening on Unix socket "/tmp/.s.PGSQL.9003"
LOG: database system was shut down at 2022-06-09 10:19:24 CEST
LOG: invalid primary checkpoint record
PANIC: could not locate a valid checkpoint record
LOG: startup process (PID 8773) was terminated by signal 6: Aborted
LOG: aborting startup due to startup process failure
LOG: database system is shut down
This issue is seen in both Windows and Linux OS platforms.
To Brief on the Scenario: Our product say example with Version A1 uses Postgres 10 and in the latest version of our product (Say A2) we upgraded the Postgres to 13.
When we try to upgrade our Product through InstallAnyWhere from A1 to A2, Postgres service fails with above mentioned error.
Could you please suggest the probable cause of the issue. Let us know if you require any more information.
Thanks,
Prabhakar
Internal Use - Confidential
Internal Use - Confidential
On Fri, 2022-06-17 at 06:35 +0000, Mahendrakar, Prabhakar - Dell Team wrote: > We are facing checkpoint related issues from PostGreSQL 13.4 ( could not locate a valid checkpoint record) and Postgresservice fails to come up. > ======================================================================================================== > LOG: starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit > LOG: listening on IPv4 address "127.0.0.1", port 9003 > LOG: listening on IPv4 address "10.xx.xx.x..", port 9003 > LOG: listening on Unix socket "/tmp/.s.PGSQL.9003" > LOG: database system was shut down at 2022-06-09 10:19:24 CEST > LOG: invalid primary checkpoint record > PANIC: could not locate a valid checkpoint record > LOG: startup process (PID 8773) was terminated by signal 6: Aborted > LOG: aborting startup due to startup process failure > LOG: database system is shut down > > This issue is seen in both Windows and Linux OS platforms. > > To Brief on the Scenario: Our product say example with Version A1 uses Postgres 10 and in the > latest version of our product (Say A2) we upgraded the Postgres to 13. > When we try to upgrade our Product through InstallAnyWhere from A1 to A2, > Postgres service fails with above mentioned error. You should not be using 13.4, but the latest minor release, at this point 13.7. Anyway, the error message looks like somebody removed the contents of the "pg_wal" directory. How exactly did you upgrade PostgreSQL? Yours, Laurenz Albe -- Cybertec | https://www.cybertec-postgresql.com
RE: Postgresql error : PANIC: could not locate a valid checkpoint record
Thanks for the response.
Yes, we have taken care of proper shut down of Postgres before initiating the Upgrade.
pg_resetwal – I have read that using pg_resetwal may cause the Database more inconsistent and should be used only as a last resort.
Also this problem ( checkpoint related issue -could not locate a valid checkpoint record ) is not happening frequently. This issue is seen with the large size of Data base.
Please let me know if you require any more information.
Thanks,
Prabhakar
From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Friday, June 17, 2022 3:39 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record
[EXTERNAL EMAIL]
Assuming you have shut down your postgres properly before upgrading, it should be safe for you to run pg_resetwal.
It should help in this case.
pt., 17 cze 2022 o 12:03 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):
Yes, We are using the pg_upgrade utility of Postgres.
From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Friday, June 17, 2022 3:31 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record
[EXTERNAL EMAIL]
Hi,
Have you done pg_upgrade post Postgres 13 installation?
Cheers,
Mateusz
pt., 17 cze 2022 o 11:20 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):
Hello,
Good Morning !
We are facing checkpoint related issues from PostGreSQL 13.4 ( could not locate a valid checkpoint record) and Postgres service fails to come up.
========================================================================================================
LOG: starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit
LOG: listening on IPv4 address "127.0.0.1", port 9003
LOG: listening on IPv4 address "10.xx.xx.x..", port 9003
LOG: listening on Unix socket "/tmp/.s.PGSQL.9003"
LOG: database system was shut down at 2022-06-09 10:19:24 CEST
LOG: invalid primary checkpoint record
PANIC: could not locate a valid checkpoint record
LOG: startup process (PID 8773) was terminated by signal 6: Aborted
LOG: aborting startup due to startup process failure
LOG: database system is shut down
This issue is seen in both Windows and Linux OS platforms.
To Brief on the Scenario: Our product say example with Version A1 uses Postgres 10 and in the latest version of our product (Say A2) we upgraded the Postgres to 13.
When we try to upgrade our Product through InstallAnyWhere from A1 to A2, Postgres service fails with above mentioned error.
Could you please suggest the probable cause of the issue. Let us know if you require any more information.
Thanks,
Prabhakar
Internal Use - Confidential
Internal Use - Confidential
Internal Use - Confidential
RE: Postgresql error : PANIC: could not locate a valid checkpoint record
Could you please provide some insights in the below query. We are in need of this as it this issue is seen at some of our customer’s site.
Thanks,
Prabhakar
Internal Use - Confidential
From: Mahendrakar, Prabhakar - Dell Team
Sent: Monday, June 20, 2022 4:22 PM
To: Mateusz Henicz
Cc: pgsql-general@lists.postgresql.org
Subject: RE: Postgresql error : PANIC: could not locate a valid checkpoint record
Thanks for the response.
Yes, we have taken care of proper shut down of Postgres before initiating the Upgrade.
pg_resetwal – I have read that using pg_resetwal may cause the Database more inconsistent and should be used only as a last resort.
Also this problem ( checkpoint related issue -could not locate a valid checkpoint record ) is not happening frequently. This issue is seen with the large size of Data base.
Please let me know if you require any more information.
Thanks,
Prabhakar
From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Friday, June 17, 2022 3:39 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record
[EXTERNAL EMAIL]
Assuming you have shut down your postgres properly before upgrading, it should be safe for you to run pg_resetwal.
It should help in this case.
pt., 17 cze 2022 o 12:03 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):
Yes, We are using the pg_upgrade utility of Postgres.
From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Friday, June 17, 2022 3:31 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record
[EXTERNAL EMAIL]
Hi,
Have you done pg_upgrade post Postgres 13 installation?
Cheers,
Mateusz
pt., 17 cze 2022 o 11:20 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):
Hello,
Good Morning !
We are facing checkpoint related issues from PostGreSQL 13.4 ( could not locate a valid checkpoint record) and Postgres service fails to come up.
========================================================================================================
LOG: starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit
LOG: listening on IPv4 address "127.0.0.1", port 9003
LOG: listening on IPv4 address "10.xx.xx.x..", port 9003
LOG: listening on Unix socket "/tmp/.s.PGSQL.9003"
LOG: database system was shut down at 2022-06-09 10:19:24 CEST
LOG: invalid primary checkpoint record
PANIC: could not locate a valid checkpoint record
LOG: startup process (PID 8773) was terminated by signal 6: Aborted
LOG: aborting startup due to startup process failure
LOG: database system is shut down
This issue is seen in both Windows and Linux OS platforms.
To Brief on the Scenario: Our product say example with Version A1 uses Postgres 10 and in the latest version of our product (Say A2) we upgraded the Postgres to 13.
When we try to upgrade our Product through InstallAnyWhere from A1 to A2, Postgres service fails with above mentioned error.
Could you please suggest the probable cause of the issue. Let us know if you require any more information.
Thanks,
Prabhakar
Internal Use - Confidential
Internal Use - Confidential
Internal Use - Confidential
About pg_resetwal, you are right that it should be done as a last resort. On the other hand, if your production database does not start after upgrade and you do not have any way to rollback your changes it might be it.
Could you please provide some insights in the below query. We are in need of this as it this issue is seen at some of our customer’s site.
Thanks,
Prabhakar
Internal Use - Confidential
From: Mahendrakar, Prabhakar - Dell Team
Sent: Monday, June 20, 2022 4:22 PM
To: Mateusz Henicz
Cc: pgsql-general@lists.postgresql.org
Subject: RE: Postgresql error : PANIC: could not locate a valid checkpoint record
Thanks for the response.
Yes, we have taken care of proper shut down of Postgres before initiating the Upgrade.
pg_resetwal – I have read that using pg_resetwal may cause the Database more inconsistent and should be used only as a last resort.
Also this problem ( checkpoint related issue -could not locate a valid checkpoint record ) is not happening frequently. This issue is seen with the large size of Data base.
Please let me know if you require any more information.
Thanks,
Prabhakar
From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Friday, June 17, 2022 3:39 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record
[EXTERNAL EMAIL]
Assuming you have shut down your postgres properly before upgrading, it should be safe for you to run pg_resetwal.
It should help in this case.
pt., 17 cze 2022 o 12:03 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):
Yes, We are using the pg_upgrade utility of Postgres.
From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Friday, June 17, 2022 3:31 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record
[EXTERNAL EMAIL]
Hi,
Have you done pg_upgrade post Postgres 13 installation?
Cheers,
Mateusz
pt., 17 cze 2022 o 11:20 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):
Hello,
Good Morning !
We are facing checkpoint related issues from PostGreSQL 13.4 ( could not locate a valid checkpoint record) and Postgres service fails to come up.
========================================================================================================
LOG: starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit
LOG: listening on IPv4 address "127.0.0.1", port 9003
LOG: listening on IPv4 address "10.xx.xx.x..", port 9003
LOG: listening on Unix socket "/tmp/.s.PGSQL.9003"
LOG: database system was shut down at 2022-06-09 10:19:24 CEST
LOG: invalid primary checkpoint record
PANIC: could not locate a valid checkpoint record
LOG: startup process (PID 8773) was terminated by signal 6: Aborted
LOG: aborting startup due to startup process failure
LOG: database system is shut down
This issue is seen in both Windows and Linux OS platforms.
To Brief on the Scenario: Our product say example with Version A1 uses Postgres 10 and in the latest version of our product (Say A2) we upgraded the Postgres to 13.
When we try to upgrade our Product through InstallAnyWhere from A1 to A2, Postgres service fails with above mentioned error.
Could you please suggest the probable cause of the issue. Let us know if you require any more information.
Thanks,
Prabhakar
Internal Use - Confidential
Internal Use - Confidential
Internal Use - Confidential
RE: Postgresql error : PANIC: could not locate a valid checkpoint record
Hello there,
Please find my response below
> You should not be using 13.4, but the latest minor release, at this point 13.7.
May be in the near future we would migrate to 13.7 and not now at present.
> Anyway, the error message looks like somebody removed the contents of the "pg_wal"
directory. How exactly did you upgrade PostgreSQL?
We are using the pg_upgrade utility for doing the upgrade.
Thanks,
Prabhakar
Internal Use - Confidential
-----Original Message-----
From: Laurenz Albe <laurenz.albe@cybertec.at>
Sent: Friday, June 17, 2022 4:10 PM
To: Mahendrakar, Prabhakar - Dell Team; pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record
[EXTERNAL EMAIL]
On Fri, 2022-06-17 at 06:35 +0000, Mahendrakar, Prabhakar - Dell Team wrote:
> We are facing checkpoint related issues from PostGreSQL 13.4 ( could not locate a valid checkpoint record) and
Postgresservice fails to come up.
> ======================================================================
> ==================================
> LOG: starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc
> (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit
> LOG: listening on IPv4 address "127.0.0.1", port 9003
> LOG: listening on IPv4 address "10.xx.xx.x..", port 9003
> LOG: listening on Unix socket "/tmp/.s.PGSQL.9003"
> LOG: database system was shut down at 2022-06-09 10:19:24 CEST
> LOG: invalid primary checkpoint record
> PANIC: could not locate a valid checkpoint record
> LOG: startup process (PID 8773) was terminated by signal 6: Aborted
> LOG: aborting startup due to startup process failure
> LOG: database system is shut down
>
> This issue is seen in both Windows and Linux OS platforms.
>
> To Brief on the Scenario: Our product say example with Version A1 uses
> Postgres 10 and in the latest version of our product (Say A2) we upgraded the Postgres to 13.
> When we try to upgrade our Product through InstallAnyWhere from A1 to
> A2, Postgres service fails with above mentioned error.
You should not be using 13.4, but the latest minor release, at this point 13.7.
Anyway, the error message looks like somebody removed the contents of the "pg_wal"
directory. How exactly did you upgrade PostgreSQL?
Yours,
Laurenz Albe
--
Cybertec |
https://urldefense.com/v3/__https://www.cybertec-postgresql.com__;!!LpKI!ihYHnSm7w4UDJHWNlSnVnxBN704J-jkctGZIQ9NPj3f9UeHxY5pPnRnpPcWN5mmrWxlXc9hVyRd8AkHaAPLaXuRtVz0WxIaDVoTQ$
[cybertec-postgresql[.]com]
RE: Postgresql error : PANIC: could not locate a valid checkpoint record
Hello there,
Please find my response as below:
- Also, are you able to reproduce the issue?
No we are not able to reproduce at our end. But this is happening occasionally at customer environment that has large DB size.
- I.e. try restoring the database to state before upgrade, try to upgrade again. Does this corruption happen again?
Could you please let us know the process to perform this operations. Can I get some document to do the same.
Thanks,
Prabhakar
From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Tuesday, June 21, 2022 5:59 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record
[EXTERNAL EMAIL]
Could you also answer on questions asked by Laurenz Albe a few emails back? That could shed some light into your issue. We would know if the upgrade was performed properly and possibly point where the issue can be.
Also, are you able to reproduce the issue? I.e. try restoring the database to state before upgrade, try to upgrade again. Does this corruption happen again?
About pg_resetwal, you are right that it should be done as a last resort. On the other hand, if your production database does not start after upgrade and you do not have any way to rollback your changes it might be it.
Cheers,
Mateusz
wt., 21 cze 2022 o 14:02 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):
Could you please provide some insights in the below query. We are in need of this as it this issue is seen at some of our customer’s site.
Thanks,
Prabhakar
Internal Use - Confidential
Internal Use - Confidential
From: Mahendrakar, Prabhakar - Dell Team
Sent: Monday, June 20, 2022 4:22 PM
To: Mateusz Henicz
Cc: pgsql-general@lists.postgresql.org
Subject: RE: Postgresql error : PANIC: could not locate a valid checkpoint record
Thanks for the response.
Yes, we have taken care of proper shut down of Postgres before initiating the Upgrade.
pg_resetwal – I have read that using pg_resetwal may cause the Database more inconsistent and should be used only as a last resort.
Also this problem ( checkpoint related issue -could not locate a valid checkpoint record ) is not happening frequently. This issue is seen with the large size of Data base.
Please let me know if you require any more information.
Thanks,
Prabhakar
From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Friday, June 17, 2022 3:39 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record
[EXTERNAL EMAIL]
Assuming you have shut down your postgres properly before upgrading, it should be safe for you to run pg_resetwal.
It should help in this case.
pt., 17 cze 2022 o 12:03 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):
Yes, We are using the pg_upgrade utility of Postgres.
From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Friday, June 17, 2022 3:31 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record
[EXTERNAL EMAIL]
Hi,
Have you done pg_upgrade post Postgres 13 installation?
Cheers,
Mateusz
pt., 17 cze 2022 o 11:20 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):
Hello,
Good Morning !
We are facing checkpoint related issues from PostGreSQL 13.4 ( could not locate a valid checkpoint record) and Postgres service fails to come up.
========================================================================================================
LOG: starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit
LOG: listening on IPv4 address "127.0.0.1", port 9003
LOG: listening on IPv4 address "10.xx.xx.x..", port 9003
LOG: listening on Unix socket "/tmp/.s.PGSQL.9003"
LOG: database system was shut down at 2022-06-09 10:19:24 CEST
LOG: invalid primary checkpoint record
PANIC: could not locate a valid checkpoint record
LOG: startup process (PID 8773) was terminated by signal 6: Aborted
LOG: aborting startup due to startup process failure
LOG: database system is shut down
This issue is seen in both Windows and Linux OS platforms.
To Brief on the Scenario: Our product say example with Version A1 uses Postgres 10 and in the latest version of our product (Say A2) we upgraded the Postgres to 13.
When we try to upgrade our Product through InstallAnyWhere from A1 to A2, Postgres service fails with above mentioned error.
Could you please suggest the probable cause of the issue. Let us know if you require any more information.
Thanks,
Prabhakar
Internal Use - Confidential
Internal Use - Confidential
Internal Use - Confidential
RE: Postgresql error : PANIC: could not locate a valid checkpoint record
Hello,
Is it possible to explicitly issue a checkpoint before we move on to the pg_upgrade command?
so that in the circumstances of the Upgrade issues (like PANIC: could not locate a valid checkpoint record), we still have this last explicit checkpoint available.
Please let us know your thoughts on this.
Thanks,
Prabhakar
Internal Use - Confidential
From: Mahendrakar, Prabhakar - Dell Team
Sent: Wednesday, June 22, 2022 4:54 PM
To: Mateusz Henicz
Cc: pgsql-general@lists.postgresql.org
Subject: RE: Postgresql error : PANIC: could not locate a valid checkpoint record
Hello there,
Please find my response as below:
- Also, are you able to reproduce the issue?
No we are not able to reproduce at our end. But this is happening occasionally at customer environment that has large DB size.
- I.e. try restoring the database to state before upgrade, try to upgrade again. Does this corruption happen again?
Could you please let us know the process to perform this operations. Can I get some document to do the same.
Thanks,
Prabhakar
From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Tuesday, June 21, 2022 5:59 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record
[EXTERNAL EMAIL]
Could you also answer on questions asked by Laurenz Albe a few emails back? That could shed some light into your issue. We would know if the upgrade was performed properly and possibly point where the issue can be.
Also, are you able to reproduce the issue? I.e. try restoring the database to state before upgrade, try to upgrade again. Does this corruption happen again?
About pg_resetwal, you are right that it should be done as a last resort. On the other hand, if your production database does not start after upgrade and you do not have any way to rollback your changes it might be it.
Cheers,
Mateusz
wt., 21 cze 2022 o 14:02 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):
Could you please provide some insights in the below query. We are in need of this as it this issue is seen at some of our customer’s site.
Thanks,
Prabhakar
Internal Use - Confidential
Internal Use - Confidential
From: Mahendrakar, Prabhakar - Dell Team
Sent: Monday, June 20, 2022 4:22 PM
To: Mateusz Henicz
Cc: pgsql-general@lists.postgresql.org
Subject: RE: Postgresql error : PANIC: could not locate a valid checkpoint record
Thanks for the response.
Yes, we have taken care of proper shut down of Postgres before initiating the Upgrade.
pg_resetwal – I have read that using pg_resetwal may cause the Database more inconsistent and should be used only as a last resort.
Also this problem ( checkpoint related issue -could not locate a valid checkpoint record ) is not happening frequently. This issue is seen with the large size of Data base.
Please let me know if you require any more information.
Thanks,
Prabhakar
From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Friday, June 17, 2022 3:39 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record
[EXTERNAL EMAIL]
Assuming you have shut down your postgres properly before upgrading, it should be safe for you to run pg_resetwal.
It should help in this case.
pt., 17 cze 2022 o 12:03 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):
Yes, We are using the pg_upgrade utility of Postgres.
From: Mateusz Henicz <mateuszhenicz@gmail.com>
Sent: Friday, June 17, 2022 3:31 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record
[EXTERNAL EMAIL]
Hi,
Have you done pg_upgrade post Postgres 13 installation?
Cheers,
Mateusz
pt., 17 cze 2022 o 11:20 Mahendrakar, Prabhakar - Dell Team <Prabhakar.Mahendraka@dellteam.com> napisał(a):
Hello,
Good Morning !
We are facing checkpoint related issues from PostGreSQL 13.4 ( could not locate a valid checkpoint record) and Postgres service fails to come up.
========================================================================================================
LOG: starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit
LOG: listening on IPv4 address "127.0.0.1", port 9003
LOG: listening on IPv4 address "10.xx.xx.x..", port 9003
LOG: listening on Unix socket "/tmp/.s.PGSQL.9003"
LOG: database system was shut down at 2022-06-09 10:19:24 CEST
LOG: invalid primary checkpoint record
PANIC: could not locate a valid checkpoint record
LOG: startup process (PID 8773) was terminated by signal 6: Aborted
LOG: aborting startup due to startup process failure
LOG: database system is shut down
This issue is seen in both Windows and Linux OS platforms.
To Brief on the Scenario: Our product say example with Version A1 uses Postgres 10 and in the latest version of our product (Say A2) we upgraded the Postgres to 13.
When we try to upgrade our Product through InstallAnyWhere from A1 to A2, Postgres service fails with above mentioned error.
Could you please suggest the probable cause of the issue. Let us know if you require any more information.
Thanks,
Prabhakar
Internal Use - Confidential
Internal Use - Confidential
Internal Use - Confidential
On Fri, Jun 24, 2022 at 01:03:57PM +0000, Mahendrakar, Prabhakar - Dell Team wrote: > Is it possible to explicitly issue a checkpoint before we move on to > the pg_upgrade command? > so that in the circumstances of the Upgrade issues (like PANIC: > could not locate a valid checkpoint record), we still have this > last explicit checkpoint available. > > Please let us know your thoughts on this. Well, you have mentioned the use of pg_upgrade, but you are giving zero details about what kind of command you used, how you handled the clusters before and after that were upgraded, or what kind of environment is getting used. With this little amount of details, nobody will be able to guess what's happening. This issue could also be caused by the environment. For example, it is possible in some carelessly-setup enviromnents that a flush is issued and recognized as completed by the OS, and thought as completed by Postgres, but an application layer between the OS and the actual hardware did not issue the flush (be it an OS, FS, disk or a VM-related thing), which would make this issue reachable. -- Michael
Attachment
RE: Postgresql error : PANIC: could not locate a valid checkpoint record
Hi Team, Hi Team, We are using the below command to perform the PG Upgrade. '/opt/XXXX/YYYY/services/datastore/engine-new/bin/pg_upgrade' -b '/opt/XXXX/YYYY/services/datastore/engine/bin' -B '/opt/XXXX/YYYY/services/datastore/engine-new/bin' -d '/opt/XXXX/YYYY/db/data' -D '/opt/XXXX/YYYY/db/data-new' -p 9003 -P 9003 -U apollosuperuser -k -j 4 -v" This is not a cluster environment and is standalone one. Could you please let us know how to restore the database to a state before proceeding to upgrade or explicitly issue a checkpointbefore we move on to the pg_upgrade command. Thanks, Prabhakar Internal Use - Confidential -----Original Message----- From: Michael Paquier <michael@paquier.xyz> Sent: Monday, June 27, 2022 5:34 AM To: Mahendrakar, Prabhakar - Dell Team Cc: Mateusz Henicz; pgsql-general@lists.postgresql.org Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint record On Fri, Jun 24, 2022 at 01:03:57PM +0000, Mahendrakar, Prabhakar - Dell Team wrote: > Is it possible to explicitly issue a checkpoint before we move on to > the pg_upgrade command? > so that in the circumstances of the Upgrade issues (like PANIC: > could not locate a valid checkpoint record), we still have this last > explicit checkpoint available. > > Please let us know your thoughts on this. Well, you have mentioned the use of pg_upgrade, but you are giving zero details about what kind of command you used, howyou handled the clusters before and after that were upgraded, or what kind of environment is getting used. With thislittle amount of details, nobody will be able to guess what's happening. This issue could also be caused by the environment. For example, it is possible in some carelessly-setup enviromnents that a flush is issued and recognized as completedby the OS, and thought as completed by Postgres, but an application layer between the OS and the actual hardwaredid not issue the flush (be it an OS, FS, disk or a VM-related thing), which would make this issue reachable. -- Michael