Thread: postgresql crushed with XLogWrite error
Hello dear list! I am using postgresql 7.3.3 .Last night the errors occured. Since then I can't start the postgresql !We are recovering from the type by now ,but what happened?Is that a known bug?What to do next time? The last change I made was that: syslog=1 log_min_error_statement log_timestamp = true stats_reset_on_server_start = false I can't think why those changes could crush the postgres! Here is the log: #============================================================================================ 2004-01-05 05:30:35 PANIC: XLogWrite: write request 15/EF40E000 is past end of log 15/EF40E000 2004-01-05 05:30:35 LOG: statement: update tyuta_rashum set chugid=521 where studentid=3587261 and maslulsignid=8014022 and c ourseid=67538; 2004-01-05 05:30:35 LOG: server process (pid 2527) was terminated by signal 6 2004-01-05 05:30:35 LOG: terminating any other active server processes 2004-01-05 05:30:35 LOG: all server processes terminated; reinitializing shared memory and semaphores 2004-01-05 05:30:35 LOG: database system was interrupted at 2004-01-05 03:42:11 IST 2004-01-05 05:30:35 LOG: checkpoint record is at 15/EF40DFC0 2004-01-05 05:30:35 LOG: redo record is at 15/EF40DFC0; undo record is at 0/0; shutdown TRUE 2004-01-05 05:30:35 LOG: next transaction id: 88539506; next oid: 88761828 2004-01-05 05:30:35 LOG: database system was not properly shut down; automatic recovery in progress 2004-01-05 05:30:35 FATAL: The database system is starting up 2004-01-05 05:30:35 FATAL: The database system is starting up 2004-01-05 05:30:35 FATAL: The database system is starting up 2004-01-05 05:30:35 LOG: ReadRecord: unexpected pageaddr 15/E640E000 in log file 21, segment 239, offset 4251648 2004-01-05 05:30:35 LOG: redo is not required 2004-01-05 05:30:35 FATAL: The database system is starting up 2004-01-05 05:30:35 FATAL: The database system is starting up 2004-01-05 05:30:38 PANIC: XLogWrite: write request 15/EF40E000 is past end of log 15/EF40E000 2004-01-05 05:30:38 LOG: startup process (pid 2529) was terminated by signal 6 2004-01-05 05:30:38 LOG: aborting startup due to startup process failure #============================================================================================ And since that, every time I am trying to start the db I get this: #============================================================================================ 2004-01-05 08:49:10 LOG: database system shutdown was interrupted at 2004-01-05 05:30:35 IST 2004-01-05 08:49:10 LOG: checkpoint record is at 15/EF40DFC0 2004-01-05 08:49:10 LOG: redo record is at 15/EF40DFC0; undo record is at 0/0; shutdown TRUE 2004-01-05 08:49:10 LOG: next transaction id: 88539506; next oid: 88761828 2004-01-05 08:49:10 LOG: database system was not properly shut down; automatic recovery in progress 2004-01-05 08:49:10 LOG: ReadRecord: unexpected pageaddr 15/E640E000 in log file 21, segment 239, offset 4251648 2004-01-05 08:49:10 LOG: redo is not required 2004-01-05 08:49:12 PANIC: XLogWrite: write request 15/EF40E000 is past end of log 15/EF40E000 2004-01-05 08:49:12 LOG: startup process (pid 2907) was terminated by signal 6 2004-01-05 08:49:12 LOG: aborting startup due to startup process failure #============================================================================================ Here is the ls -al from the pg_clog directory. I can't see any problems in log file 21: -rw------- 1 postgres postgres 262144 Jul 6 2003 0000 -rw------- 1 postgres postgres 262144 Jul 10 02:10 0001 -rw------- 1 postgres postgres 262144 Jul 16 00:24 0002 -rw------- 1 postgres postgres 262144 Jul 21 21:15 0003 -rw------- 1 postgres postgres 262144 Jul 27 11:53 0004 -rw------- 1 postgres postgres 262144 Jul 29 11:11 0005 -rw------- 1 postgres postgres 262144 Jul 30 18:41 0006 -rw------- 1 postgres postgres 262144 Aug 3 20:24 0007 -rw------- 1 postgres postgres 262144 Aug 5 02:07 0008 -rw------- 1 postgres postgres 262144 Aug 6 18:52 0009 -rw------- 1 postgres postgres 262144 Aug 11 17:51 000A -rw------- 1 postgres postgres 262144 Aug 12 18:50 000B -rw------- 1 postgres postgres 262144 Aug 14 02:34 000C -rw------- 1 postgres postgres 262144 Aug 15 13:40 000D -rw------- 1 postgres postgres 262144 Aug 17 18:04 000E -rw------- 1 postgres postgres 262144 Aug 20 15:58 000F -rw------- 1 postgres postgres 262144 Aug 24 10:09 0010 -rw------- 1 postgres postgres 262144 Aug 25 14:52 0011 -rw------- 1 postgres postgres 262144 Aug 27 15:15 0012 -rw------- 1 postgres postgres 262144 Aug 31 14:58 0013 -rw------- 1 postgres postgres 262144 Sep 8 10:05 0014 -rw------- 1 postgres postgres 262144 Sep 8 13:55 0015 -rw------- 1 postgres postgres 262144 Sep 8 23:46 0016 -rw------- 1 postgres postgres 262144 Sep 9 21:01 0017 -rw------- 1 postgres postgres 262144 Sep 10 02:00 0018 -rw------- 1 postgres postgres 262144 Sep 10 09:49 0019 -rw------- 1 postgres postgres 262144 Sep 10 12:16 001A -rw------- 1 postgres postgres 262144 Sep 10 16:28 001B -rw------- 1 postgres postgres 262144 Sep 11 04:51 001C -rw------- 1 postgres postgres 262144 Sep 11 14:00 001D -rw------- 1 postgres postgres 262144 Sep 14 13:57 001E -rw------- 1 postgres postgres 262144 Sep 15 02:02 001F -rw------- 1 postgres postgres 262144 Sep 15 10:41 0020 -rw------- 1 postgres postgres 262144 Sep 15 12:37 0021 -rw------- 1 postgres postgres 262144 Sep 15 15:19 0022 -rw------- 1 postgres postgres 262144 Sep 15 22:58 0023 -rw------- 1 postgres postgres 262144 Sep 16 13:26 0024 -rw------- 1 postgres postgres 262144 Sep 17 06:43 0025 -rw------- 1 postgres postgres 262144 Sep 17 15:27 0026 -rw------- 1 postgres postgres 262144 Sep 18 13:14 0027 -rw------- 1 postgres postgres 262144 Sep 21 13:37 0028 -rw------- 1 postgres postgres 262144 Sep 22 09:21 0029 -rw------- 1 postgres postgres 262144 Sep 22 15:44 002A -rw------- 1 postgres postgres 262144 Sep 23 01:15 002B -rw------- 1 postgres postgres 262144 Sep 23 10:49 002C -rw------- 1 postgres postgres 262144 Sep 23 14:36 002D -rw------- 1 postgres postgres 262144 Sep 24 00:32 002E -rw------- 1 postgres postgres 262144 Sep 24 11:56 002F -rw------- 1 postgres postgres 262144 Sep 24 22:18 0030 -rw------- 1 postgres postgres 262144 Sep 25 14:11 0031 -rw------- 1 postgres postgres 262144 Sep 29 12:43 0032 -rw------- 1 postgres postgres 262144 Sep 30 00:31 0033 -rw------- 1 postgres postgres 262144 Sep 30 18:49 0034 -rw------- 1 postgres postgres 262144 Oct 2 14:04 0035 -rw------- 1 postgres postgres 262144 Oct 19 08:42 0036 -rw------- 1 postgres postgres 262144 Oct 19 12:59 0037 -rw------- 1 postgres postgres 262144 Oct 19 16:11 0038 -rw------- 1 postgres postgres 262144 Oct 20 10:18 0039 -rw------- 1 postgres postgres 262144 Oct 20 14:40 003A -rw------- 1 postgres postgres 262144 Oct 21 09:03 003B -rw------- 1 postgres postgres 262144 Oct 21 15:06 003C -rw------- 1 postgres postgres 262144 Oct 22 11:47 003D -rw------- 1 postgres postgres 262144 Oct 22 23:09 003E -rw------- 1 postgres postgres 262144 Oct 23 16:15 003F -rw------- 1 postgres postgres 262144 Oct 26 11:09 0040 -rw------- 1 postgres postgres 262144 Oct 26 15:16 0041 -rw------- 1 postgres postgres 262144 Oct 27 11:00 0042 -rw------- 1 postgres postgres 262144 Oct 27 15:42 0043 -rw------- 1 postgres postgres 262144 Oct 28 10:45 0044 -rw------- 1 postgres postgres 262144 Oct 28 19:55 0045 -rw------- 1 postgres postgres 262144 Oct 29 13:31 0046 -rw------- 1 postgres postgres 262144 Oct 30 11:04 0047 -rw------- 1 postgres postgres 262144 Oct 31 15:34 0048 -rw------- 1 postgres postgres 262144 Nov 2 15:27 0049 -rw------- 1 postgres postgres 262144 Nov 3 16:16 004A -rw------- 1 postgres postgres 262144 Nov 4 18:06 004B -rw------- 1 postgres postgres 262144 Nov 6 08:59 004C -rw------- 1 postgres postgres 262144 Nov 9 10:24 004D -rw------- 1 postgres postgres 262144 Nov 10 12:55 004E -rw------- 1 postgres postgres 262144 Nov 11 16:06 004F -rw------- 1 postgres postgres 262144 Nov 12 19:03 0050 -rw------- 1 postgres postgres 262144 Nov 13 15:45 0051 -rw------- 1 postgres postgres 262144 Nov 20 12:14 0052 -rw------- 1 postgres postgres 262144 Dec 17 19:56 0053 -rw------- 1 postgres postgres 122880 Jan 5 00:08 0054
On Mon, Jan 05, 2004 at 02:14:37PM +0200, Tsirkin Evgeny wrote:
> Hello dear list!
> I am using postgresql 7.3.3 .Last night the errors occured.
^
> by now ,but what happened?Is that a known bug?What to do next time?
Yes, it's a known bug, and is the very reason 7.3.4 was released.
Upgrade.
A
--
----
Andrew Sullivan 204-4141 Yonge Street
Afilias Canada Toronto, Ontario Canada
<andrew@libertyrms.info> M2P 2A8
+1 416 646 3304 x110
Thanks! Now another two questions: Is there any official bug report on this that i can show to my boss? Should i upgrade to 7.3.4 to fix that bug or it is better to get newly version? What i need is to restore my corrupted db from the tape ,upgrade postgresql (say from rpm) and just start the server.I don't want to do all the dump - restore thing. On Mon, 5 Jan 2004 08:49:13 -0500, Andrew Sullivan <andrew@libertyrms.info> wrote: > On Mon, Jan 05, 2004 at 02:14:37PM +0200, Tsirkin Evgeny wrote: >> Hello dear list! >> I am using postgresql 7.3.3 .Last night the errors occured. > ^ > >> by now ,but what happened?Is that a known bug?What to do next time? > > Yes, it's a known bug, and is the very reason 7.3.4 was released. > Upgrade. > > A >
On Mon, Jan 05, 2004 at 04:07:34PM +0200, Tsirkin Evgeny wrote:
> Thanks!
> Now another two questions:
> Is there any official bug report on this that i can show to
> my boss?
Yes, it's in the release notes. IIRC, there were like 24 or 48 hours
between the releases because of this. Tom Lane occasionally beats
himself up about the bug on-list.
> Should i upgrade to 7.3.4 to fix that bug or it is better to get newly
> version?
You can upgrade to the lastest 7.3.x release by just dropping the new
binary in place, IIRC -- read the release notes. In any case, you'll
need to get some version of 7.3 running first.
A
----
Andrew Sullivan 204-4141 Yonge Street
Afilias Canada Toronto, Ontario Canada
<andrew@libertyrms.info> M2P 2A8
+1 416 646 3304 x110
Probably a dumm question ,but it is better to ask then be sorry. Nothing is changed in config files ,so i am extractiing the binary from rpm ,copy it to the place and start the server,right? And thanks again! On Mon, 5 Jan 2004 09:09:55 -0500, Andrew Sullivan <andrew@libertyrms.info> wrote: > On Mon, Jan 05, 2004 at 04:07:34PM +0200, Tsirkin Evgeny wrote: >> Thanks! >> Now another two questions: >> Is there any official bug report on this that i can show to >> my boss? > > Yes, it's in the release notes. IIRC, there were like 24 or 48 hours > between the releases because of this. Tom Lane occasionally beats > himself up about the bug on-list. > >> Should i upgrade to 7.3.4 to fix that bug or it is better to get newly >> version? > > You can upgrade to the lastest 7.3.x release by just dropping the new > binary in place, IIRC -- read the release notes. In any case, you'll > need to get some version of 7.3 running first. > > A > > ---- > Andrew Sullivan 204-4141 Yonge Street > Afilias Canada Toronto, Ontario Canada > <andrew@libertyrms.info> M2P 2A8 > +1 416 646 3304 x110 > > > ---------------------------(end of broadcast)--------------------------- > TIP 4: Don't 'kill -9' the postmaster
On Mon, Jan 05, 2004 at 04:48:00PM +0200, Tsirkin Evgeny wrote:
> Probably a dumm question ,but it is better to ask then be sorry.
> Nothing is changed in config files ,so i am extractiing
> the binary from rpm ,copy it to the place and start the server,right?
> And thanks again!
If you're upgrading via RPM, why not just do a regular RPM
installation? Doesn't that work? (Extracting binaries from RPMs and
doing other such things sounds like the way to break something.)
A
--
----
Andrew Sullivan 204-4141 Yonge Street
Afilias Canada Toronto, Ontario Canada
<andrew@libertyrms.info> M2P 2A8
+1 416 646 3304 x110
On Mon, 5 Jan 2004 11:23:08 -0500, Andrew Sullivan <andrew@libertyrms.info> wrote: I want to keep my configs.Of course i can save my postgresql.conf (and other files that i changed and don't currently remember) to another location and then put it back in place of what rpm will put ,but as i said ,i don't actually remember what i have changed :( > > If you're upgrading via RPM, why not just do a regular RPM > installation? Doesn't that work? (Extracting binaries from RPMs and > doing other such things sounds like the way to break something.) > > A >
On Mon, Jan 05, 2004 at 07:35:16PM +0200, Tsirkin Evgeny wrote:
> On Mon, 5 Jan 2004 11:23:08 -0500, Andrew Sullivan
> <andrew@libertyrms.info> wrote:
>
> I want to keep my configs.Of course i can save my postgresql.conf
> (and other files that i changed and don't currently remember)
> to another location and then put it back in place of what rpm will
> put ,but as i said ,i don't actually remember what i have changed :(
The only things that should make a difference to you should be in the
/etc directory, assuming the RPM follows the filesystem standard (I
believe it does). My suggestion is to back everything up, then
replace the RPM in the standard way, and then compare the files one
at a time to see what's different. I'll bet the binaries and
libraries, of course, and things under /etc/[postgres?]/, and that's
it. Most likely, it'll be postgresql.conf and pg_hba.conf, and maybe
pg_ident.conf.
A
--
----
Andrew Sullivan 204-4141 Yonge Street
Afilias Canada Toronto, Ontario Canada
<andrew@libertyrms.info> M2P 2A8
+1 416 646 3304 x110
On Mon, 5 Jan 2004 12:39:20 -0500, Andrew Sullivan <andrew@libertyrms.info> wrote: > > The only things that should make a difference to you should be in the > /etc directory, assuming the RPM follows the filesystem standard (I > believe it does). It does not ,rpm puts the staff in /var/lib > My suggestion is to back everything up, then > replace the RPM in the standard way, and then compare the files one > at a time to see what's different. I'll bet the binaries and > libraries, of course, and things under /etc/[postgres?]/, and that's > it. Most likely, it'll be postgresql.conf and pg_hba.conf, and maybe > pg_ident.conf. > > A > Well, this is what i eventually did ,it was just more convinient to wget the rpms ,type rpm -Uhv postgres*rpm and then replace 2 files. (It took some work to recall what i have changed in the config files,thought) That was simpler then rpm2cpio etc... as i thought at the beginning. The server works fine by now,hope this XLogWrite thing will not come back . Thanks for the help. Evgeny
andrew@libertyrms.info (Andrew Sullivan) writes: > On Mon, Jan 05, 2004 at 07:35:16PM +0200, Tsirkin Evgeny wrote: >> On Mon, 5 Jan 2004 11:23:08 -0500, Andrew Sullivan >> <andrew@libertyrms.info> wrote: >> >> I want to keep my configs.Of course i can save my postgresql.conf >> (and other files that i changed and don't currently remember) >> to another location and then put it back in place of what rpm will >> put ,but as i said ,i don't actually remember what i have changed :( > > The only things that should make a difference to you should be in the > /etc directory, assuming the RPM follows the filesystem standard (I > believe it does). My suggestion is to back everything up, then > replace the RPM in the standard way, and then compare the files one > at a time to see what's different. I'll bet the binaries and > libraries, of course, and things under /etc/[postgres?]/, and that's > it. Most likely, it'll be postgresql.conf and pg_hba.conf, and maybe > pg_ident.conf. It's easy enough to check what files the RPM intends to update... bash-2.05a$ rpm -q -p -l netpbm-devel-9.24-3.i386.rpm /usr/include/pam.h /usr/include/pammap.h /usr/include/pbm.h /usr/include/pbmshhopt.h /usr/include/pgm.h /usr/include/pm.h ... other files omitted ... That can diminish how much is in the "everything" that needs to be backed up... -- let name="cbbrowne" and tld="libertyrms.info" in String.concat "@" [name;tld];; <http://dev6.int.libertyrms.com/> Christopher Browne (416) 646 3304 x124 (land)