Thread: Postgres database is not logging any messages after the reboot
Hi The server that hosts our production postgres databases had to be rebooted last night as the file systems got remounted as read only. After the reboot we re-started the databases but now nothing is getting logged into the pglogs. Can you please assist us in figuring out the problem and help resolve it. Below listed are the error messages in the /var/log/messages file due to which we had to reboot the server. This is a linux x86-64 server. Your prompt response will be highly appreciated. Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1): ext3_free_blocks_sb: bit already cleared for b lock 3931168 Oct 10 06:45:02 sandbox kernel: Aborting journal on device sdb1. Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1): ext3_free_blocks_sb: bit already cleared for b lock 3931169 Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1): ext3_free_blocks_sb: bit already cleared for b lock 3931170 Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1): ext3_free_blocks_sb: bit already cleared for b lock 3931171 Oct 10 06:45:02 sandbox kernel: ext3_abort called. Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1): ext3_journal_start_sb: Detected aborted journa l Oct 10 06:45:02 sandbox kernel: Remounting filesystem read-only Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in ext3_free_blocks_sb: Journal has aborted Oct 10 06:45:02 sandbox last message repeated 70 times Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write: Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in ext3_truncate: Journal has aborted Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write: Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in ext3_orphan_del: Journal has aborted Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write: Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in ext3_delete_inode: Journal has aborted Oct 10 06:45:02 sandbox kernel: __journal_remove_journal_head: freeing b_committed_data Thanks Paramjeet Bhella
Re: Postgres database is not logging any messages after the reboot
From
"Bhella Paramjeet-PFCW67"
Date:
Here are the messages from the pg_log before and after server reboot. 2007-10-10 06:22:50 CDT LOG: archived transaction log file "000000010000007000000090" 2007-10-10 06:29:04 CDT LOG: autovacuum: processing database "nsc" 2007-10-10 06:34:58 CDT LOG: archived transaction log file "000000010000007000000091" 2007-10-10 07:11:48 CDT LOG: database system was interrupted at 2007-10-10 06:40:05 CDT 2007-10-10 07:11:48 CDT LOG: checkpoint record is at 70/925E7B68 2007-10-10 07:11:48 CDT LOG: redo record is at 70/925E7B68; undo record is at 0/0; shutdown FALSE 2007-10-10 07:11:48 CDT LOG: next transaction ID: 656073787; next OID: 2915367 2007-10-10 07:11:48 CDT LOG: next MultiXactId: 3599; next MultiXactOffset: 7197 2007-10-10 07:11:48 CDT LOG: database system was not properly shut down; automatic recovery in progress 2007-10-10 07:11:48 CDT LOG: redo starts at 70/925E7BB8 nsc 2007-10-10 07:11:48 CDT attbase FATAl: the database system is starting up nsc 2007-10-10 07:11:48 CDT attbase FATAl: the database system is starting up nsc 2007-10-10 07:11:48 CDT attbase FATAl: the database system is starting up nsc 2007-10-10 07:11:48 CDT attbase FATAl: the database system is starting up nsc 2007-10-10 07:11:48 CDT attbase FATAl: the database system is starting up : : : nsc 2007-10-10 07:11:53 CDT vrzbase FATAl: the database system is starting up nsc 2007-10-10 07:11:53 CDT vrzbase FATAl: the database system is starting up 2007-10-10 07:11:53 CDT LOG: record with zero length at 70/92BBF1C8 2007-10-10 07:11:53 CDT LOG: redo done at 70/92BBF198 nsc 2007-10-10 07:11:53 CDT vrzbase FATAl: the database system is starting up nsc 2007-10-10 07:11:53 CDT vrzbase FATAl: the database system is starting up nsc 2007-10-10 07:11:53 CDT vrzbase FATAl: the database system is starting up 2007-10-10 07:11:53 CDT LOG: database system is ready 2007-10-10 07:11:53 CDT LOG: transaction ID wrap limit is 2147484146, limited by database "postgres" 2007-10-10 07:11:59 CDT LOG: archived transaction log file "000000010000007000000092" nsc 2007-10-10 07:19:37 CDT attbase WARNING: there is no transaction in progress nsc 2007-10-10 07:20:48 CDT attbase WARNING: there is no transaction in progress nsc 2007-10-10 07:23:53 CDT attbase WARNING: there is no transaction in progress nsc 2007-10-10 07:24:09 CDT vrzbase LOG: unexpected EOF on client connection nsc 2007-10-10 07:24:09 CDT vrzbase LOG: unexpected EOF on client connection nsc 2007-10-10 07:24:09 CDT vrzbase LOG: unexpected EOF on client connection nsc 2007-10-10 07:24:09 CDT vrzbase LOG: unexpected EOF on client connection Thanks Paramjeet Kaur -----Original Message----- From: Bhella Paramjeet-PFCW67 Sent: Wednesday, October 10, 2007 10:43 AM To: 'pgsql-admin@postgresql.org' Cc: Subbiah Stalin-XCGF84 Subject: Postgres database is not logging any messages after the reboot Hi The server that hosts our production postgres databases had to be rebooted last night as the file systems got remounted as read only. After the reboot we re-started the databases but now nothing is getting logged into the pglogs. Can you please assist us in figuring out the problem and help resolve it. Below listed are the error messages in the /var/log/messages file due to which we had to reboot the server. This is a linux x86-64 server. Your prompt response will be highly appreciated. Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1): ext3_free_blocks_sb: bit already cleared for b lock 3931168 Oct 10 06:45:02 sandbox kernel: Aborting journal on device sdb1. Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1): ext3_free_blocks_sb: bit already cleared for b lock 3931169 Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1): ext3_free_blocks_sb: bit already cleared for b lock 3931170 Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1): ext3_free_blocks_sb: bit already cleared for b lock 3931171 Oct 10 06:45:02 sandbox kernel: ext3_abort called. Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1): ext3_journal_start_sb: Detected aborted journa l Oct 10 06:45:02 sandbox kernel: Remounting filesystem read-only Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in ext3_free_blocks_sb: Journal has aborted Oct 10 06:45:02 sandbox last message repeated 70 times Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write: Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in ext3_truncate: Journal has aborted Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write: Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in ext3_orphan_del: Journal has aborted Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write: Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in ext3_delete_inode: Journal has aborted Oct 10 06:45:02 sandbox kernel: __journal_remove_journal_head: freeing b_committed_data Thanks Paramjeet Bhella
"Bhella Paramjeet-PFCW67" <PBhella@Motorola.com> writes:
> Here are the messages from the pg_log before and after server reboot.
Uh, I don't see the problem? That looks like a perfectly normal
crash-recovery process, and it certainly doesn't seem to be the case
that "nothing is getting logged". What were you expecting to see?
regards, tom lane
Re: Postgres database is not logging any messages after the reboot
From
"Bhella Paramjeet-PFCW67"
Date:
Hi Tom
After the server reboot nothing is getting logged by the postgres
database into pg_log like messages about archived log, autovacuum,
panic, alerts etc. The alerts from the pglogs get forwarded to our cell
phones and since no messages are getting logged we will not get notified
in case of a problem with database. This is the first time we have seen
this strange behaviour. Your help in resolving the issue will be highly
appreciated.
Thanks
Paramjeet Kaur
-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
Sent: Thursday, October 11, 2007 7:30 AM
To: Bhella Paramjeet-PFCW67
Cc: pgsql-admin@postgresql.org; Subbiah Stalin-XCGF84
Subject: Re: [ADMIN] Postgres database is not logging any messages after
the reboot
"Bhella Paramjeet-PFCW67" <PBhella@Motorola.com> writes:
> Here are the messages from the pg_log before and after server reboot.
Uh, I don't see the problem? That looks like a perfectly normal
crash-recovery process, and it certainly doesn't seem to be the case
that "nothing is getting logged". What were you expecting to see?
regards, tom lane
"Bhella Paramjeet-PFCW67" <PBhella@Motorola.com> writes:
> After the server reboot nothing is getting logged by the postgres
> database into pg_log like messages about archived log, autovacuum,
> panic, alerts etc.
Sounds to me like you changed your logging settings...
regards, tom lane
Bhella Paramjeet-PFCW67 wrote: > Hi > > The server that hosts our production postgres databases had to be > rebooted last night as the file systems got remounted as read only. > After the reboot we re-started the databases but now nothing is getting > logged into the pglogs. Can you please assist us in figuring out the > problem and help resolve it. Below listed are the error messages in the > /var/log/messages file due to which we had to reboot the server. This is > a linux x86-64 server. Your prompt response will be highly appreciated. When a filesystem is automatically remounted read-only by the operating system, it's because serious trouble is ahead. Rebooting to remount read-write seems a recipe for disaster unless you have corrected the problem. This may be a Linux kernel bug, a disk/cable/controller failure, or something else. -- Alvaro Herrera Valdivia, Chile ICBM: S 39º 49' 18.1", W 73º 13' 56.4" "The ability to monopolize a planet is insignificant next to the power of the source"