Thread: FATAL: the database system is starting up

FATAL: the database system is starting up

From
"William E. Moreno A."
Date:
Solution: http://archives.postgresql.org/pgsql-general/2007-12/msg01339.php
Solution: Message-id: <476D6DE1.4050600@latnet.lv>
<text/plain>
 
Problem: FATAL:  the database system is starting up
 
Solved:   change postgresql_flags in /etc/rc.conf to: postgresql_flags="-s -m fast" or postgresql_flags="-s -m smart"
 
Operatin System: FreeBSD 8.2 p4

Database:            PostgreSQL Server 8.2.9_2

I have had a lot of work trying to solve this issue, and I finally found the solution.
Why at this time, with the latest version of PostgreSQL 8.4.9_2 this isssue continue ?
I knew by the logs, that many people have the same or similar problem.
 
Another issue that I have with this version 8.4.9_2 of PostgreSQL. Is about pgsql user that is not created in a clean
install,via ports. PostgreSQL version 8.3 creates pgsql user, but not version 8.4. 
 
Thanks Very Much
 
I'm happy with the solution found and waiting for the other about the pgsql user creation in version 8.4.9_2

P.D.: El presente correo no contiene tildes para evitarle al receptor
inconvenientes en su lectura.

Re: FATAL: the database system is starting up

From
patrick keshishian
Date:
On Wed, Oct 26, 2011 at 4:49 PM, William E. Moreno A.
<wmoreno3@yahoo.com> wrote:
> Solution: http://archives.postgresql.org/pgsql-general/2007-12/msg01339.php
> Solution: Message-id: <476D6DE1.4050600@latnet.lv>
> <text/plain>
>
> Problem: FATAL:  the database system is starting up
>
> Solved:   change postgresql_flags in /etc/rc.conf to: postgresql_flags="-s -m fast" or postgresql_flags="-s -m smart"

interesting enough, got bit by this recently. Our set up does not have
a "-w" option. During upgrades we "cycle" our PostgreSQL daemon. We
had to change the "pg_ctl stop" command to "-mimmediate" because
during upgrades we would find that someone had an abandoned psql shell
running for days and that would halt our upgrade script.

Adding "-m immediate" for shutdown seemed like a logical choice to get
around this sort of a "procedural" issue(s).

But I see this in our PostgreSQL log (warning: gmail will wrap lines):

Oct 26 15:41:53 foo postgres[20168]: [51-1] NOTICE:  ALTER TABLE / ADD
PRIMARY KEY ...
Oct 26 15:48:42 foo postgres[19909]: [1-1] LOG:  received immediate
shutdown request
Oct 26 15:48:44 foo postgres[21919]: [1-1] LOG:  database system was
interrupted at 2011-10-26 15:45:21 GMT
Oct 26 15:48:44 foo postgres[21919]: [2-1] LOG:  checkpoint record is
at 10/8C21F898
Oct 26 15:48:44 foo postgres[21919]: [3-1] LOG:  redo record is at
10/8C21F898; undo record is at 0/0; shutdown FALSE
Oct 26 15:48:44 foo postgres[21919]: [4-1] LOG:  next transaction ID:
176547132; next OID: 40216935
Oct 26 15:48:44 foo postgres[21919]: [5-1] LOG:  database system was
not properly shut down; automatic recovery in progress
Oct 26 15:48:44 foo postgres[21920]: [1-1] FATAL:  the database system
is starting up
Oct 26 15:48:45 foo postgres[21919]: [6-1] LOG:  redo starts at 10/8C21F8D8
Oct 26 15:48:45 foo postgres[21919]: [7-1] LOG:  record with zero
length at 10/8C2D1D38
Oct 26 15:48:45 foo postgres[21919]: [8-1] LOG:  redo done at 10/8C2D1D14
Oct 26 15:48:46 foo postgres[21922]: [1-1] FATAL:  the database system
is starting up
Oct 26 15:48:47 foo postgres[21925]: [1-1] FATAL:  the database system
is starting up
Oct 26 15:48:48 foo postgres[21930]: [1-1] FATAL:  the database system
is starting up
Oct 26 15:48:48 foo postgres[21919]: [9-1] LOG:  recycled transaction
log file "000000100000008B"
Oct 26 15:48:48 foo postgres[21919]: [10-1] LOG:  database system is ready

[above output is from a PG7.4.x, but have noticed similar behavior in PG9.1.1]

In order to retain the "immediate" option for shutdown, I added a
three second sleep in our upgrade script after the "cycle" command.

Kinda hacky but ...

--patrick




> Operatin System: FreeBSD 8.2 p4
>
> Database:            PostgreSQL Server 8.2.9_2
>
> I have had a lot of work trying to solve this issue, and I finally found the solution.
> Why at this time, with the latest version of PostgreSQL 8.4.9_2 this isssue continue ?
> I knew by the logs, that many people have the same or similar problem.
>
> Another issue that I have with this version 8.4.9_2 of PostgreSQL. Is about pgsql user that is not created in a clean
install,via ports. PostgreSQL version 8.3 creates pgsql user, but not version 8.4. 
>
> Thanks Very Much
>
> I'm happy with the solution found and waiting for the other about the pgsql user creation in version 8.4.9_2
>
> P.D.: El presente correo no contiene tildes para evitarle al receptor
> inconvenientes en su lectura.
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>

Re: FATAL: the database system is starting up

From
Scott Marlowe
Date:
On Wed, Oct 26, 2011 at 6:09 PM, patrick keshishian <pkeshish@gmail.com> wrote:
> On Wed, Oct 26, 2011 at 4:49 PM, William E. Moreno A.
> <wmoreno3@yahoo.com> wrote:
>> Solution: http://archives.postgresql.org/pgsql-general/2007-12/msg01339.php
>> Solution: Message-id: <476D6DE1.4050600@latnet.lv>
>> <text/plain>
>>
>> Problem: FATAL:  the database system is starting up
>>
>> Solved:   change postgresql_flags in /etc/rc.conf to: postgresql_flags="-s -m fast" or postgresql_flags="-s -m
smart"
>
> interesting enough, got bit by this recently. Our set up does not have
> a "-w" option. During upgrades we "cycle" our PostgreSQL daemon. We
> had to change the "pg_ctl stop" command to "-mimmediate" because
> during upgrades we would find that someone had an abandoned psql shell
> running for days and that would halt our upgrade script.
>
> Adding "-m immediate" for shutdown seemed like a logical choice to get
> around this sort of a "procedural" issue(s).

didn't -m fast work?

Re: FATAL: the database system is starting up

From
patrick keshishian
Date:
On Wed, Oct 26, 2011 at 5:46 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
> On Wed, Oct 26, 2011 at 6:09 PM, patrick keshishian <pkeshish@gmail.com> wrote:
>> On Wed, Oct 26, 2011 at 4:49 PM, William E. Moreno A.
>> <wmoreno3@yahoo.com> wrote:
>>> Solution: http://archives.postgresql.org/pgsql-general/2007-12/msg01339.php
>>> Solution: Message-id: <476D6DE1.4050600@latnet.lv>
>>> <text/plain>
>>>
>>> Problem: FATAL:  the database system is starting up
>>>
>>> Solved:   change postgresql_flags in /etc/rc.conf to: postgresql_flags="-s -m fast" or postgresql_flags="-s -m
smart"
>>
>> interesting enough, got bit by this recently. Our set up does not have
>> a "-w" option. During upgrades we "cycle" our PostgreSQL daemon. We
>> had to change the "pg_ctl stop" command to "-mimmediate" because
>> during upgrades we would find that someone had an abandoned psql shell
>> running for days and that would halt our upgrade script.
>>
>> Adding "-m immediate" for shutdown seemed like a logical choice to get
>> around this sort of a "procedural" issue(s).
>
> didn't -m fast work?

I forget now why "-m fast" wasn't used. I tried to dig up anything I
had in my notes and did a set of new experiments (hence the late
reply), but didn't come up with much.

About the only difference with '-m fast' and '-m immediate' seems to
be the following log entry:

 .... database system was
not properly shut down; automatic recovery in progress

For now, I've changed the script to use '-m fast' and removed the
sleep. Will find out after some moderate use of it whether it was a
wise decision or not :-)

--patrick

Re: FATAL: the database system is starting up

From
patrick keshishian
Date:
On Tue, Nov 1, 2011 at 5:43 AM, patrick keshishian <pkeshish@gmail.com> wrote:
> On Wed, Oct 26, 2011 at 5:46 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
>> On Wed, Oct 26, 2011 at 6:09 PM, patrick keshishian <pkeshish@gmail.com> wrote:
>>> On Wed, Oct 26, 2011 at 4:49 PM, William E. Moreno A.
>>> <wmoreno3@yahoo.com> wrote:
>>>> Solution: http://archives.postgresql.org/pgsql-general/2007-12/msg01339.php
>>>> Solution: Message-id: <476D6DE1.4050600@latnet.lv>
>>>> <text/plain>
>>>>
>>>> Problem: FATAL:  the database system is starting up
>>>>
>>>> Solved:   change postgresql_flags in /etc/rc.conf to: postgresql_flags="-s -m fast" or postgresql_flags="-s -m
smart"
>>>
>>> interesting enough, got bit by this recently. Our set up does not have
>>> a "-w" option. During upgrades we "cycle" our PostgreSQL daemon. We
>>> had to change the "pg_ctl stop" command to "-mimmediate" because
>>> during upgrades we would find that someone had an abandoned psql shell
>>> running for days and that would halt our upgrade script.
>>>
>>> Adding "-m immediate" for shutdown seemed like a logical choice to get
>>> around this sort of a "procedural" issue(s).
>>
>> didn't -m fast work?
>
> I forget now why "-m fast" wasn't used. I tried to dig up anything I
> had in my notes and did a set of new experiments (hence the late
> reply), but didn't come up with much.
>
> About the only difference with '-m fast' and '-m immediate' seems to
> be the following log entry:
>
>  .... database system was
> not properly shut down; automatic recovery in progress
>
> For now, I've changed the script to use '-m fast' and removed the
> sleep. Will find out after some moderate use of it whether it was a
> wise decision or not :-)

(just for the archives ...)

So, even though 'stop -m fast' does in fact work nicer than "-m
immediate", the sleep(3) is required after a "pg_ctl -D $PGDATA start"
command, in our scripts.

Excuse gmail line-wraps:

# /etc/init.d/postgres stop
Stopping PostgreSQL Database (smart)... OK
# date ; /etc/init.d/postgres start ; date ; i=0 ; while : ; do psql
-lU postgres 2>/dev/null && date && break ; i=$((i+1)) ; printf "PG9
connect try #%d\n" $i ; done
Mon Nov  7 20:58:06 PST 2011
Starting PostgreSQL Database... OK
Mon Nov  7 20:58:06 PST 2011
PG9 connect try #1
PG9 connect try #2
PG9 connect try #3
PG9 connect try #4
PG9 connect try #5
PG9 connect try #6
PG9 connect try #7
PG9 connect try #8
PG9 connect try #9
PG9 connect try #10
PG9 connect try #11
PG9 connect try #12
PG9 connect try #13
PG9 connect try #14
PG9 connect try #15
PG9 connect try #16
PG9 connect try #17
PG9 connect try #18
PG9 connect try #19
PG9 connect try #20
PG9 connect try #21
PG9 connect try #22
PG9 connect try #23
PG9 connect try #24
PG9 connect try #25
PG9 connect try #26
PG9 connect try #27
PG9 connect try #28
PG9 connect try #29
PG9 connect try #30
PG9 connect try #31
PG9 connect try #32
PG9 connect try #33
PG9 connect try #34
PG9 connect try #35
PG9 connect try #36
PG9 connect try #37
PG9 connect try #38
PG9 connect try #39
PG9 connect try #40
PG9 connect try #41
PG9 connect try #42
PG9 connect try #43
PG9 connect try #44
PG9 connect try #45
PG9 connect try #46
PG9 connect try #47
PG9 connect try #48
PG9 connect try #49
PG9 connect try #50
PG9 connect try #51
PG9 connect try #52
PG9 connect try #53
PG9 connect try #54
PG9 connect try #55
                                  List of databases
   Name    |  Owner   | Encoding |  Collation  |    Ctype    |
Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
 foo       | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
=c/postgres          +
           |          |          |             |             |
postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
=c/postgres          +
           |          |          |             |             |
postgres=CTc/postgres
 x1        | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
(5 rows)

Mon Nov  7 20:58:07 PST 2011
# /etc/init.d/postgres stop
Stopping PostgreSQL Database (smart)... OK
# date ; /etc/init.d/postgres start ; date ; i=0 ; while : ; do psql
-lU postgres 2>/dev/null && date && break ; i=$((i+1)) ; printf "PG9
connect try #%d\n" $i ; done
Mon Nov  7 20:58:49 PST 2011
Starting PostgreSQL Database... OK
Mon Nov  7 20:58:49 PST 2011
PG9 connect try #1
PG9 connect try #2
PG9 connect try #3
PG9 connect try #4
PG9 connect try #5
PG9 connect try #6
PG9 connect try #7
PG9 connect try #8
PG9 connect try #9
PG9 connect try #10
PG9 connect try #11
PG9 connect try #12
PG9 connect try #13
PG9 connect try #14
PG9 connect try #15
PG9 connect try #16
PG9 connect try #17
PG9 connect try #18
PG9 connect try #19
PG9 connect try #20
PG9 connect try #21
PG9 connect try #22
PG9 connect try #23
PG9 connect try #24
PG9 connect try #25
PG9 connect try #26
PG9 connect try #27
PG9 connect try #28
PG9 connect try #29
PG9 connect try #30
PG9 connect try #31
PG9 connect try #32
PG9 connect try #33
PG9 connect try #34
PG9 connect try #35
PG9 connect try #36
PG9 connect try #37
PG9 connect try #38
PG9 connect try #39
PG9 connect try #40
PG9 connect try #41
PG9 connect try #42
PG9 connect try #43
PG9 connect try #44
                                  List of databases
   Name    |  Owner   | Encoding |  Collation  |    Ctype    |
Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
 foo       | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
=c/postgres          +
           |          |          |             |             |
postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
=c/postgres          +
           |          |          |             |             |
postgres=CTc/postgres
 x1        | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
(5 rows)

Mon Nov  7 20:58:50 PST 2011


vs same with PG7:

# date ; /etc/init.d/postgres-db start ; date ; i=0 ; while : ; do
i=$((i+1)) ; psql -lU postgres 2>/dev/null && date && break ; printf
"PG7 connect try #%d\n" $i ; done
Mon Nov  7 21:57:54 PST 2011
Starting PostgreSQL Database... OK
Mon Nov  7 21:57:54 PST 2011
PG7 connect try #1
PG7 connect try #2
        List of databases
   Name    |  Owner   | Encoding
-----------+----------+-----------
 foo       | postgres | SQL_ASCII
 template0 | postgres | SQL_ASCII
 template1 | postgres | SQL_ASCII
(3 rows)

Mon Nov  7 21:57:54 PST 2011
# /etc/init.d/postgres-db stop
Stopping PostgreSQL Database (smart)... OK
# date ; /etc/init.d/postgres-db start ; date ; i=0 ; while : ; do
i=$((i+1)) ; psql -lU postgres 2>/dev/null && date && break ; printf
"PG7 connect try #%d\n" $i ; done
Mon Nov  7 21:59:35 PST 2011
Starting PostgreSQL Database... OK
Mon Nov  7 21:59:35 PST 2011
PG7 connect try #1
PG7 connect try #2
        List of databases
   Name    |  Owner   | Encoding
-----------+----------+-----------
 foo       | postgres | SQL_ASCII
 template0 | postgres | SQL_ASCII
 template1 | postgres | SQL_ASCII
(3 rows)

Mon Nov  7 21:59:35 PST 2011


The "cute" thing with PG7 is, we also start the contrib/pg_autovacuum
in "/etc/init.d/postgre start", which would "cover up" for, what seems
to be two connect failures right after start of PG7.

Cheers,
--patrick

Re: FATAL: the database system is starting up

From
David Johnston
Date:
fluxh wrote
> I have the same problem. I have a log like patrick keshishian. I have not
> a backup.
>
> I don't know that do.
>
> Help me please!!!
>
> Information:
>
> - Server with RAID1 on SAS HD Hot-plug.
> - Ubuntu Server 12.04 x64.
> - Postgresql 9.1.
>
> When I do "psql" PostgreSQL shows "psql: FATAL: the database system is
> starting up" and when I write pg_dump or I try connect with pgAdmin3.

You should supply your actual log file and also provide some narrative of
what exactly you were doing prior to this condition occurring.  Some context
as to architecture wouldn't hurt - such as how many and what kinds of
applications are connecting to the database, especially those that maintain
persistent connections.

System commad output like top and ps may also be of value though after you
provide the background a more directed Q&A can follow.

In short it sounds like you need to force PostgreSQL to shutdown and then
start it back up again.  If you are using the apt package you'd do this via
some invocation of pg_ctlcluster for which there is ample documentation of
both it and the underlying pg_ctl command.

David J.




--
View this message in context:
http://postgresql.1045698.n5.nabble.com/FATAL-the-database-system-is-starting-up-tp4941646p5795847.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.


Re: FATAL: the database system is starting up

From
fluxh
Date:
I have the same problem. I have a log like patrick keshishian. I have not a
backup.

I don't know that do.

Help me please!!!

Information:

- Server with RAID1 on SAS HD Hot-plug.
- Ubuntu Server 12.04 x64.
- Postgresql 9.1.

When I do "psql" PostgreSQL shows "psql: FATAL: the database system is
starting up" and when I write pg_dump or I try connect with pgAdmin3.



--
View this message in context:
http://postgresql.1045698.n5.nabble.com/FATAL-the-database-system-is-starting-up-tp4941646p5795841.html
Sent from the PostgreSQL - general mailing list archive at Nabble.com.