Thread: Postgres database is not logging any messages after the reboot

Postgres database is not logging any messages after the reboot

From
"Bhella Paramjeet-PFCW67"
Date:
Hi

The server that hosts our production postgres databases had to be
rebooted last night as the file systems got remounted as read only.
After the reboot we re-started the databases but now nothing is getting
logged into the pglogs. Can you please assist us in figuring out the
problem and help resolve it. Below listed are the error messages in the
/var/log/messages file due to which we had to reboot the server. This is
a linux x86-64 server. Your prompt response will be highly appreciated.

 Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b
lock 3931168
Oct 10 06:45:02 sandbox kernel: Aborting journal on device sdb1.
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b
lock 3931169
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b
lock 3931170
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b
lock 3931171
Oct 10 06:45:02 sandbox kernel: ext3_abort called.
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_journal_start_sb: Detected aborted journa
l
Oct 10 06:45:02 sandbox kernel: Remounting filesystem read-only
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_free_blocks_sb: Journal has aborted
Oct 10 06:45:02 sandbox last message repeated 70 times
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_reserve_inode_write: Journal has aborte
d
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_truncate: Journal has aborted
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_reserve_inode_write: Journal has aborte
d
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_orphan_del: Journal has aborted
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_reserve_inode_write: Journal has aborte
d
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_delete_inode: Journal has aborted
Oct 10 06:45:02 sandbox kernel: __journal_remove_journal_head: freeing
b_committed_data

Thanks
Paramjeet Bhella


Re: Postgres database is not logging any messages after the reboot

From
"Bhella Paramjeet-PFCW67"
Date:
Here are the messages from the pg_log before and after server reboot.

2007-10-10 06:22:50 CDT  LOG:  archived transaction log file
"000000010000007000000090"
 2007-10-10 06:29:04 CDT  LOG:  autovacuum: processing database "nsc"
 2007-10-10 06:34:58 CDT  LOG:  archived transaction log file
"000000010000007000000091"
 2007-10-10 07:11:48 CDT  LOG:  database system was interrupted at
2007-10-10 06:40:05 CDT
 2007-10-10 07:11:48 CDT  LOG:  checkpoint record is at 70/925E7B68
 2007-10-10 07:11:48 CDT  LOG:  redo record is at 70/925E7B68; undo
record is at 0/0; shutdown FALSE
 2007-10-10 07:11:48 CDT  LOG:  next transaction ID: 656073787; next
OID: 2915367
 2007-10-10 07:11:48 CDT  LOG:  next MultiXactId: 3599; next
MultiXactOffset: 7197
 2007-10-10 07:11:48 CDT  LOG:  database system was not properly shut
down; automatic recovery in progress
 2007-10-10 07:11:48 CDT  LOG:  redo starts at 70/925E7BB8
nsc 2007-10-10 07:11:48 CDT attbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl:  the database system is
starting up
:
:
:
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl:  the database system is
starting up
 2007-10-10 07:11:53 CDT  LOG:  record with zero length at 70/92BBF1C8
 2007-10-10 07:11:53 CDT  LOG:  redo done at 70/92BBF198
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl:  the database system is
starting up
 2007-10-10 07:11:53 CDT  LOG:  database system is ready
 2007-10-10 07:11:53 CDT  LOG:  transaction ID wrap limit is 2147484146,
limited by database "postgres"
 2007-10-10 07:11:59 CDT  LOG:  archived transaction log file
"000000010000007000000092"
nsc 2007-10-10 07:19:37 CDT attbase WARNING:  there is no transaction in
progress
nsc 2007-10-10 07:20:48 CDT attbase WARNING:  there is no transaction in
progress
nsc 2007-10-10 07:23:53 CDT attbase WARNING:  there is no transaction in
progress
nsc 2007-10-10 07:24:09 CDT vrzbase LOG:  unexpected EOF on client
connection
nsc 2007-10-10 07:24:09 CDT vrzbase LOG:  unexpected EOF on client
connection
nsc 2007-10-10 07:24:09 CDT vrzbase LOG:  unexpected EOF on client
connection
nsc 2007-10-10 07:24:09 CDT vrzbase LOG:  unexpected EOF on client
connection

Thanks
Paramjeet Kaur

-----Original Message-----
From: Bhella Paramjeet-PFCW67
Sent: Wednesday, October 10, 2007 10:43 AM
To: 'pgsql-admin@postgresql.org'
Cc: Subbiah Stalin-XCGF84
Subject: Postgres database is not logging any messages after the reboot

Hi

The server that hosts our production postgres databases had to be
rebooted last night as the file systems got remounted as read only.
After the reboot we re-started the databases but now nothing is getting
logged into the pglogs. Can you please assist us in figuring out the
problem and help resolve it. Below listed are the error messages in the
/var/log/messages file due to which we had to reboot the server. This is
a linux x86-64 server. Your prompt response will be highly appreciated.

 Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931168 Oct 10
06:45:02 sandbox kernel: Aborting journal on device sdb1.
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931169 Oct 10
06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931170 Oct 10
06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931171 Oct 10
06:45:02 sandbox kernel: ext3_abort called.
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_journal_start_sb: Detected aborted journa l Oct 10 06:45:02 sandbox
kernel: Remounting filesystem read-only Oct 10 06:45:02 sandbox kernel:
EXT3-fs error (device sdb1) in ext3_free_blocks_sb: Journal has aborted
Oct 10 06:45:02 sandbox last message repeated 70 times Oct 10 06:45:02
sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write:
Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error
(device sdb1) in ext3_truncate: Journal has aborted Oct 10 06:45:02
sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write:
Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error
(device sdb1) in ext3_orphan_del: Journal has aborted Oct 10 06:45:02
sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write:
Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error
(device sdb1) in ext3_delete_inode: Journal has aborted Oct 10 06:45:02
sandbox kernel: __journal_remove_journal_head: freeing b_committed_data

Thanks
Paramjeet Bhella


Re: Postgres database is not logging any messages after the reboot

From
Tom Lane
Date:
"Bhella Paramjeet-PFCW67" <PBhella@Motorola.com> writes:
> Here are the messages from the pg_log before and after server reboot.

Uh, I don't see the problem?  That looks like a perfectly normal
crash-recovery process, and it certainly doesn't seem to be the case
that "nothing is getting logged".  What were you expecting to see?

            regards, tom lane

Re: Postgres database is not logging any messages after the reboot

From
"Bhella Paramjeet-PFCW67"
Date:
Hi Tom

After the server reboot nothing is getting logged by the postgres
database into pg_log like messages about archived log, autovacuum,
panic, alerts etc. The alerts from the pglogs get forwarded to our cell
phones and since no messages are getting logged we will not get notified
in case of a problem with database. This is the first time we have seen
this strange behaviour. Your help in resolving the issue will be highly
appreciated.

Thanks
Paramjeet Kaur

-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Thursday, October 11, 2007 7:30 AM
To: Bhella Paramjeet-PFCW67
Cc: pgsql-admin@postgresql.org; Subbiah Stalin-XCGF84
Subject: Re: [ADMIN] Postgres database is not logging any messages after
the reboot

"Bhella Paramjeet-PFCW67" <PBhella@Motorola.com> writes:
> Here are the messages from the pg_log before and after server reboot.

Uh, I don't see the problem?  That looks like a perfectly normal
crash-recovery process, and it certainly doesn't seem to be the case
that "nothing is getting logged".  What were you expecting to see?

            regards, tom lane

Re: Postgres database is not logging any messages after the reboot

From
Tom Lane
Date:
"Bhella Paramjeet-PFCW67" <PBhella@Motorola.com> writes:
> After the server reboot nothing is getting logged by the postgres
> database into pg_log like messages about archived log, autovacuum,
> panic, alerts etc.

Sounds to me like you changed your logging settings...

            regards, tom lane

Re: Postgres database is not logging any messages after the reboot

From
Alvaro Herrera
Date:
Bhella Paramjeet-PFCW67 wrote:
> Hi
>
> The server that hosts our production postgres databases had to be
> rebooted last night as the file systems got remounted as read only.
> After the reboot we re-started the databases but now nothing is getting
> logged into the pglogs. Can you please assist us in figuring out the
> problem and help resolve it. Below listed are the error messages in the
> /var/log/messages file due to which we had to reboot the server. This is
> a linux x86-64 server. Your prompt response will be highly appreciated.

When a filesystem is automatically remounted read-only by the operating
system, it's because serious trouble is ahead.  Rebooting to remount
read-write seems a recipe for disaster unless you have corrected the
problem.

This may be a Linux kernel bug, a disk/cable/controller failure, or
something else.

--
Alvaro Herrera       Valdivia, Chile   ICBM: S 39º 49' 18.1", W 73º 13' 56.4"
"The ability to monopolize a planet is insignificant
next to the power of the source"