Thread: Upgrade Process Says "The database server was not shut downcleanly" but it was
I'm in the process to update my Postgres 10 to 12 (it's my first server
upgrade). I've shutdown my Postgres 10 using "systemctl stop
postgresql-10.service" and "systemctl status postgresql-10.service" says:
Active: inactive (dead) since Sun 2020-05-03 22:51:46 CEST;
After that I've started the upgrade process with:
/usr/pgsql-12/bin/pg_upgrade --old-bindir=/usr/pgsql-10/bin/
--new-bindir=/usr/pgsql-12/bin/ --old-datadir=/var/lib/pgsql/10/data
--new-datadir=/var/lib/pgsql/12/data --old-options '-c
config_file=/var/lib/pgsql/10/data/postgresql.conf' --new-options '-c
config_file=/var/lib/pgsql/12/data/postgresql.conf'
And the output is:
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for tables WITH OIDS ok
Checking for invalid "sql_identifier" user columns ok
Creating dump of global objects ok
Creating dump of database schemas
ok
Checking for presence of required libraries ok
Checking database user is the install user ok
Checking for prepared transactions ok
If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.
Performing Upgrade
------------------
Analyzing all rows in the new cluster ok
Freezing all rows in the new cluster ok
Deleting files from new pg_xact ok
Copying old pg_xact to new server ok
Setting next transaction ID and epoch for new cluster ok
Deleting files from new pg_multixact/offsets ok
Copying old pg_multixact/offsets to new server ok
Deleting files from new pg_multixact/members ok
Copying old pg_multixact/members to new server ok
Setting next multixact ID and offset for new cluster ok
Resetting WAL archives ok
Setting frozenxid and minmxid counters in new cluster ok
Restoring global objects in the new cluster ok
Restoring database schemas in the new cluster
ok
Copying user relation files
ok
Setting next OID for new cluster
*failure*
Consult the last few lines of "pg_upgrade_utility.log" for
the probable cause of the failure.
Failure, exiting
The pg_upgrade_utility.log shows:
command: "/usr/pgsql-12/bin/pg_resetwal" -o 1091293 "/var/lib/pgsql/12/data"
>> "pg_upgrade_utility.log" 2>&1
The database server was not shut down cleanly.
Resetting the write-ahead log might cause data to be lost.
If you want to proceed anyway, use -f to force reset
1. I did shutdown the server properly "systemctl stop
postgresql-10.service", is there a better way to do that?
2. Where do I set the "-f" flag if I choose to force reset?
--
Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html
Re: Upgrade Process Says "The database server was not shut downcleanly" but it was
From
Adrian Klaver
Date:
On 5/3/20 2:53 PM, TalGloz wrote: > I'm in the process to update my Postgres 10 to 12 (it's my first server > upgrade). I've shutdown my Postgres 10 using "systemctl stop > postgresql-10.service" and "systemctl status postgresql-10.service" says: > Active: inactive (dead) since Sun 2020-05-03 22:51:46 CEST; > > After that I've started the upgrade process with: > /usr/pgsql-12/bin/pg_upgrade --old-bindir=/usr/pgsql-10/bin/ > --new-bindir=/usr/pgsql-12/bin/ --old-datadir=/var/lib/pgsql/10/data > --new-datadir=/var/lib/pgsql/12/data --old-options '-c > config_file=/var/lib/pgsql/10/data/postgresql.conf' --new-options '-c > config_file=/var/lib/pgsql/12/data/postgresql.conf' > > And the output is: > Performing Consistency Checks > ----------------------------- > Checking cluster versions ok > Checking database user is the install user ok > Checking database connection settings ok > Checking for prepared transactions ok > Checking for reg* data types in user tables ok > Checking for contrib/isn with bigint-passing mismatch ok > Checking for tables WITH OIDS ok > Checking for invalid "sql_identifier" user columns ok > Creating dump of global objects ok > Creating dump of database schemas > > ok > Checking for presence of required libraries ok > Checking database user is the install user ok > Checking for prepared transactions ok > > If pg_upgrade fails after this point, you must re-initdb the > new cluster before continuing. > > Performing Upgrade > ------------------ > Analyzing all rows in the new cluster ok > Freezing all rows in the new cluster ok > Deleting files from new pg_xact ok > Copying old pg_xact to new server ok > Setting next transaction ID and epoch for new cluster ok > Deleting files from new pg_multixact/offsets ok > Copying old pg_multixact/offsets to new server ok > Deleting files from new pg_multixact/members ok > Copying old pg_multixact/members to new server ok > Setting next multixact ID and offset for new cluster ok > Resetting WAL archives ok > Setting frozenxid and minmxid counters in new cluster ok > Restoring global objects in the new cluster ok > Restoring database schemas in the new cluster > > ok > Copying user relation files > > ok > Setting next OID for new cluster > *failure* > > Consult the last few lines of "pg_upgrade_utility.log" for > the probable cause of the failure. > Failure, exiting > > The pg_upgrade_utility.log shows: > command: "/usr/pgsql-12/bin/pg_resetwal" -o 1091293 "/var/lib/pgsql/12/data" Looks like you did not shutdown the 12 instance. >>> "pg_upgrade_utility.log" 2>&1 > The database server was not shut down cleanly. > Resetting the write-ahead log might cause data to be lost. > If you want to proceed anyway, use -f to force reset > > 1. I did shutdown the server properly "systemctl stop > postgresql-10.service", is there a better way to do that? > 2. Where do I set the "-f" flag if I choose to force reset? > > > > -- > Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html > > -- Adrian Klaver adrian.klaver@aklaver.com
Re: Upgrade Process Says "The database server was not shut downcleanly" but it was
From
TalGloz
Date:
Adrian Klaver-4 wrote > On 5/3/20 2:53 PM, > > Looks like you did not shutdown the 12 instance. > > -- > Adrian Klaver > adrian.klaver@ This is weird because "systemctl staus postgresql-12.service" shows that the service is dead. Best regards, TalGloz -- Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html
Re: Upgrade Process Says "The database server was not shut downcleanly" but it was
From
Adrian Klaver
Date:
On 5/3/20 3:11 PM, TalGloz wrote: > Adrian Klaver-4 wrote >> On 5/3/20 2:53 PM, >> >> Looks like you did not shutdown the 12 instance. >> >> -- >> Adrian Klaver > >> adrian.klaver@ > > > This is weird because "systemctl staus postgresql-12.service" shows that the > service is dead. It is now: "The database server was not shut down cleanly." Before running pg_upgrade again I would:\ 1) Start the 12 instance to make sure it will start cleanly. 2) Shut it and the 10 instance down. 3) Try the pg_upgrade again this time with: https://www.postgresql.org/docs/12/pgupgrade.html --check check clusters only, don't change any data 4) Assuming 3) completes, repeat w/o --check > > Best regards, > TalGloz > > > > -- > Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html > > -- Adrian Klaver adrian.klaver@aklaver.com
Re: Upgrade Process Says "The database server was not shut downcleanly" but it was
From
TalGloz
Date:
Adrian Klaver-4 wrote > On 5/3/20 3:11 PM, TalGloz wrote: >> Adrian Klaver-4 wrote >>> On 5/3/20 2:53 PM, >>> >>> Looks like you did not shutdown the 12 instance. >>> >>> -- >>> Adrian Klaver >> >>> adrian.klaver@ >> >> >> This is weird because "systemctl staus postgresql-12.service" shows that >> the >> service is dead. > > It is now: > "The database server was not shut down cleanly." > > Before running pg_upgrade again I would:\ > > 1) Start the 12 instance to make sure it will start cleanly. > > 2) Shut it and the 10 instance down. > > 3) Try the pg_upgrade again this time with: > > https://www.postgresql.org/docs/12/pgupgrade.html > > --check > > check clusters only, don't change any data > > 4) Assuming 3) completes, repeat w/o --check > >> >> Best regards, >> TalGloz >> >> >> >> -- >> Sent from: >> https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html >> >> > > > -- > Adrian Klaver > adrian.klaver@ Thanks I'll try your steps tommorow. What I meant to say is that "systemctl staus postgresql-12.service" was always showing that the service is dead, even before I've started the upgrade process. Best regards, TalGloz -- Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html
Re: Upgrade Process Says "The database server was not shut downcleanly" but it was
From
Adrian Klaver
Date:
On 5/3/20 3:33 PM, TalGloz wrote: > Adrian Klaver-4 wrote >> On 5/3/20 3:11 PM, TalGloz wrote: >>> Adrian Klaver-4 wrote >>>> On 5/3/20 2:53 PM, >>>> >>>> Looks like you did not shutdown the 12 instance. >>>> >>>> -- >>>> Adrian Klaver >>> >>>> adrian.klaver@ >>> >>> >>> This is weird because "systemctl staus postgresql-12.service" shows that >>> the >>> service is dead. >> >> It is now: >> "The database server was not shut down cleanly." >> >> Before running pg_upgrade again I would:\ >> >> 1) Start the 12 instance to make sure it will start cleanly. >> >> 2) Shut it and the 10 instance down. >> >> 3) Try the pg_upgrade again this time with: >> >> https://www.postgresql.org/docs/12/pgupgrade.html >> >> --check >> >> check clusters only, don't change any data >> >> 4) Assuming 3) completes, repeat w/o --check >> >>> >>> Best regards, >>> TalGloz >>> >>> >>> >>> -- >>> Sent from: >>> https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html >>> >>> >> >> >> -- >> Adrian Klaver > >> adrian.klaver@ > > > Thanks I'll try your steps tommorow. What I meant to say is that "systemctl > staus postgresql-12.service" was always showing that the service is dead, > even before I've started the upgrade process. Hmm, then maybe the unclean shutdown was from a previous use. At at any rate you passed this point in the process: "If pg_upgrade fails after this point, you must re-initdb the new cluster before continuing." You are going to need to start with a fresh cluster anyway. > > Best regards, > TalGloz > > > > -- > Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html > > -- Adrian Klaver adrian.klaver@aklaver.com
Re: Upgrade Process Says "The database server was not shut downcleanly" but it was
From
TalGloz
Date:
Well I tried your steps, both Postgres 10 and 12 are in perfect running conditions and were shut down using the "systemctl" command. When I execute: /usr/pgsql-12/bin/pg_upgrade --old-bindir=/usr/pgsql-10/bin/ --new-bindir=/usr/pgsql-12/bin/ --old-datadir=/var/lib/pgsql/10/data --new-datadir=/var/lib/pgsql/12/data --old-options '-c config_file=/var/lib/pgsql/10/data/postgresql.conf' --new-options '-c config_file=/var/lib/pgsql/12/data/postgresql.conf' --check I get: Performing Consistency Checks ----------------------------- Checking cluster versions ok Checking database user is the install user ok Checking database connection settings ok Checking for prepared transactions ok Checking for reg* data types in user tables ok Checking for contrib/isn with bigint-passing mismatch ok Checking for tables WITH OIDS ok Checking for invalid "sql_identifier" user columns ok Checking for presence of required libraries ok Checking database user is the install user ok Checking for prepared transactions ok *Clusters are compatible* But if I execute the same command right after that without the "--check" flag I get: Performing Consistency Checks ----------------------------- Checking cluster versions ok The target cluster was not shut down cleanly. Failure, exiting Both of the servers were still shut down. If I start form the beginning with a fresh Prostgres 12 installation and execute the upgrade command right away without the "--check" flag I get the problem described in my original post. Best regards, TalGlo -- Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html
Re: Upgrade Process Says "The database server was not shut downcleanly" but it was
From
Adrian Klaver
Date:
On 5/11/20 2:45 PM, TalGloz wrote: > Well I tried your steps, both Postgres 10 and 12 are in perfect running > conditions and were shut down using the "systemctl" command. When I execute: > > /usr/pgsql-12/bin/pg_upgrade --old-bindir=/usr/pgsql-10/bin/ > --new-bindir=/usr/pgsql-12/bin/ --old-datadir=/var/lib/pgsql/10/data > --new-datadir=/var/lib/pgsql/12/data --old-options '-c > config_file=/var/lib/pgsql/10/data/postgresql.conf' --new-options '-c > config_file=/var/lib/pgsql/12/data/postgresql.conf' --check The --*-options are redundant as you have already pointed pg_upgrade at them with --*-datadir. Not sure it has anything to do with this, but you might want to take them out just to be sure. Have you run: ps ax| grep post before and between steps to make sure there is not a Postgres instance running? > > I get: > > Performing Consistency Checks > ----------------------------- > Checking cluster versions ok > Checking database user is the install user ok > Checking database connection settings ok > Checking for prepared transactions ok > Checking for reg* data types in user tables ok > Checking for contrib/isn with bigint-passing mismatch ok > Checking for tables WITH OIDS ok > Checking for invalid "sql_identifier" user columns ok > Checking for presence of required libraries ok > Checking database user is the install user ok > Checking for prepared transactions ok > > *Clusters are compatible* > > But if I execute the same command right after that without the "--check" > flag I get: > > Performing Consistency Checks > ----------------------------- > Checking cluster versions ok > > The target cluster was not shut down cleanly. > Failure, exiting > > > Both of the servers were still shut down. If I start form the beginning with > a fresh Prostgres 12 installation and execute the upgrade command right away > without the "--check" flag I get the problem described in my original post. > > Best regards, > TalGlo > > > > -- > Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html > > -- Adrian Klaver adrian.klaver@aklaver.com
Re: Upgrade Process Says "The database server was not shut downcleanly" but it was
From
Adrian Klaver
Date:
On 5/11/20 2:45 PM, TalGloz wrote: > Well I tried your steps, both Postgres 10 and 12 are in perfect running > conditions and were shut down using the "systemctl" command. When I execute: > > /usr/pgsql-12/bin/pg_upgrade --old-bindir=/usr/pgsql-10/bin/ > --new-bindir=/usr/pgsql-12/bin/ --old-datadir=/var/lib/pgsql/10/data > --new-datadir=/var/lib/pgsql/12/data --old-options '-c > config_file=/var/lib/pgsql/10/data/postgresql.conf' --new-options '-c > config_file=/var/lib/pgsql/12/data/postgresql.conf' --check > > I get: > > Performing Consistency Checks > ----------------------------- > Checking cluster versions ok > Checking database user is the install user ok > Checking database connection settings ok > Checking for prepared transactions ok > Checking for reg* data types in user tables ok > Checking for contrib/isn with bigint-passing mismatch ok > Checking for tables WITH OIDS ok > Checking for invalid "sql_identifier" user columns ok > Checking for presence of required libraries ok > Checking database user is the install user ok > Checking for prepared transactions ok > > *Clusters are compatible* > > But if I execute the same command right after that without the "--check" > flag I get: > > Performing Consistency Checks > ----------------------------- > Checking cluster versions ok > > The target cluster was not shut down cleanly. > Failure, exiting > > > Both of the servers were still shut down. If I start form the beginning with > a fresh Prostgres 12 installation and execute the upgrade command right away > without the "--check" flag I get the problem described in my original post. How are you installing Postgres? Are the 10 and 12 packages coming from the same location? > > Best regards, > TalGlo > > > > -- > Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html > > -- Adrian Klaver adrian.klaver@aklaver.com
Re: Upgrade Process Says "The database server was not shut downcleanly" but it was
From
TalGloz
Date:
Adrian Klaver-4 wrote > On 5/11/20 2:45 PM, TalGloz wrote: >> Well I tried your steps, both Postgres 10 and 12 are in perfect running >> conditions and were shut down using the "systemctl" command. When I >> execute: >> >> /usr/pgsql-12/bin/pg_upgrade --old-bindir=/usr/pgsql-10/bin/ >> --new-bindir=/usr/pgsql-12/bin/ --old-datadir=/var/lib/pgsql/10/data >> --new-datadir=/var/lib/pgsql/12/data --old-options '-c >> config_file=/var/lib/pgsql/10/data/postgresql.conf' --new-options '-c >> config_file=/var/lib/pgsql/12/data/postgresql.conf' --check >> >> I get: >> >> Performing Consistency Checks >> ----------------------------- >> Checking cluster versions >> ok >> Checking database user is the install user ok >> Checking database connection settings ok >> Checking for prepared transactions ok >> Checking for reg* data types in user tables ok >> Checking for contrib/isn with bigint-passing mismatch ok >> Checking for tables WITH OIDS ok >> Checking for invalid "sql_identifier" user columns ok >> Checking for presence of required libraries ok >> Checking database user is the install user ok >> Checking for prepared transactions ok >> >> *Clusters are compatible* >> >> But if I execute the same command right after that without the "--check" >> flag I get: >> >> Performing Consistency Checks >> ----------------------------- >> Checking cluster versions ok >> >> The target cluster was not shut down cleanly. >> Failure, exiting >> >> >> Both of the servers were still shut down. If I start form the beginning >> with >> a fresh Prostgres 12 installation and execute the upgrade command right >> away >> without the "--check" flag I get the problem described in my original >> post. > > How are you installing Postgres? > > Are the 10 and 12 packages coming from the same location? > >> >> Best regards, >> TalGlo >> >> >> >> -- >> Sent from: >> https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html >> >> > > > -- > Adrian Klaver > adrian.klaver@ This is my output for "ps ax| grep post" it is always the same after shunting down the services. 11232 pts/0 S+ 0:00 grep --color=auto post I've used the same installation steps for both of the instances: 1. Add the PostgreSQL Yum Repository (needed to be done once) 2. Install the desired Postgres instance using "sudo dnf install postgresqlXX-*" Best regards, TalGloz -- Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html
Re: Upgrade Process Says "The database server was not shut downcleanly" but it was
From
Adrian Klaver
Date:
On 5/12/20 2:51 PM, TalGloz wrote: > Adrian Klaver-4 wrote >> On 5/11/20 2:45 PM, TalGloz wrote: >>> Well I tried your steps, both Postgres 10 and 12 are in perfect running >>> conditions and were shut down using the "systemctl" command. When I >>> execute: >>> >>> /usr/pgsql-12/bin/pg_upgrade --old-bindir=/usr/pgsql-10/bin/ >>> --new-bindir=/usr/pgsql-12/bin/ --old-datadir=/var/lib/pgsql/10/data >>> --new-datadir=/var/lib/pgsql/12/data --old-options '-c >>> config_file=/var/lib/pgsql/10/data/postgresql.conf' --new-options '-c >>> config_file=/var/lib/pgsql/12/data/postgresql.conf' --check >>> >>> I get: >>> >>> Performing Consistency Checks >>> ----------------------------- >>> Checking cluster versions >>> ok >>> Checking database user is the install user ok >>> Checking database connection settings ok >>> Checking for prepared transactions ok >>> Checking for reg* data types in user tables ok >>> Checking for contrib/isn with bigint-passing mismatch ok >>> Checking for tables WITH OIDS ok >>> Checking for invalid "sql_identifier" user columns ok >>> Checking for presence of required libraries ok >>> Checking database user is the install user ok >>> Checking for prepared transactions ok >>> >>> *Clusters are compatible* >>> >>> But if I execute the same command right after that without the "--check" >>> flag I get: >>> >>> Performing Consistency Checks >>> ----------------------------- >>> Checking cluster versions ok >>> >>> The target cluster was not shut down cleanly. >>> Failure, exiting >>> >>> >>> Both of the servers were still shut down. If I start form the beginning >>> with >>> a fresh Prostgres 12 installation and execute the upgrade command right >>> away >>> without the "--check" flag I get the problem described in my original >>> post. >> >> How are you installing Postgres? >> >> Are the 10 and 12 packages coming from the same location? >> >>> >>> Best regards, >>> TalGlo >>> >>> >>> >>> -- >>> Sent from: >>> https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html >>> >>> >> >> >> -- >> Adrian Klaver > >> adrian.klaver@ > > > > This is my output for "ps ax| grep post" it is always the same after > shunting down the services. > 11232 pts/0 S+ 0:00 grep --color=auto post > > I've used the same installation steps for both of the instances: > > 1. Add the PostgreSQL Yum Repository (needed to be done once) > 2. Install the desired Postgres instance using "sudo dnf install > postgresqlXX-*" Have you tried the upgrade w/o the --options switches? Do the regular Postgres logs show anything? Have the systemctl scripts been changed? And wild hunch from another thread: "Is there a significant difference in Memory settings between 9.5 and 12" In particular this message: https://www.postgresql.org/message-id/CAEaSS0YDpb_L1Ve%2B4aNhNz7nSN5M%2BEpT9fV8AUuEYnY2tDtuDw%40mail.gmail.com See bottom of message for what OP says about sysctl. > > Best regards, > TalGloz > > > > -- > Sent from: https://www.postgresql-archive.org/PostgreSQL-general-f1843780.html > > -- Adrian Klaver adrian.klaver@aklaver.com