Thread: postgresql crushed with XLogWrite error

postgresql crushed with XLogWrite error

From
Tsirkin Evgeny
Date:
  Hello dear list!
  I am using postgresql 7.3.3 .Last night the errors occured.
  Since then I can't start the postgresql !We are recovering from the type
  by now ,but what happened?Is that a known bug?What to do next time?
  The last change I made was that:
  syslog=1
  log_min_error_statement
  log_timestamp = true
  stats_reset_on_server_start = false
  I can't think why those changes could crush the postgres!

  Here is the log:

  #============================================================================================
  2004-01-05 05:30:35 PANIC:  XLogWrite: write request 15/EF40E000 is past
  end of log 15/EF40E000
  2004-01-05 05:30:35 LOG:  statement: update tyuta_rashum set chugid=521
  where studentid=3587261 and maslulsignid=8014022 and c
  ourseid=67538;
  2004-01-05 05:30:35 LOG:  server process (pid 2527) was terminated by
  signal 6
  2004-01-05 05:30:35 LOG:  terminating any other active server processes
  2004-01-05 05:30:35 LOG:  all server processes terminated;
  reinitializing shared memory and semaphores
  2004-01-05 05:30:35 LOG:  database system was interrupted at 2004-01-05
  03:42:11 IST
  2004-01-05 05:30:35 LOG:  checkpoint record is at 15/EF40DFC0
  2004-01-05 05:30:35 LOG:  redo record is at 15/EF40DFC0; undo record is
  at 0/0; shutdown TRUE
  2004-01-05 05:30:35 LOG:  next transaction id: 88539506; next oid:
  88761828
  2004-01-05 05:30:35 LOG:  database system was not properly shut down;
  automatic recovery in progress
  2004-01-05 05:30:35 FATAL:  The database system is starting up
  2004-01-05 05:30:35 FATAL:  The database system is starting up
  2004-01-05 05:30:35 FATAL:  The database system is starting up
  2004-01-05 05:30:35 LOG:  ReadRecord: unexpected pageaddr 15/E640E000 in
  log file 21, segment 239, offset 4251648
  2004-01-05 05:30:35 LOG:  redo is not required
  2004-01-05 05:30:35 FATAL:  The database system is starting up
  2004-01-05 05:30:35 FATAL:  The database system is starting up
  2004-01-05 05:30:38 PANIC:  XLogWrite: write request 15/EF40E000 is past
  end of log 15/EF40E000
  2004-01-05 05:30:38 LOG:  startup process (pid 2529) was terminated by
  signal 6
  2004-01-05 05:30:38 LOG:  aborting startup due to startup process failure
  #============================================================================================

  And since that, every time I am trying to start the db I get this:

  #============================================================================================
  2004-01-05 08:49:10 LOG:  database system shutdown was interrupted at
  2004-01-05 05:30:35 IST
  2004-01-05 08:49:10 LOG:  checkpoint record is at 15/EF40DFC0
  2004-01-05 08:49:10 LOG:  redo record is at 15/EF40DFC0; undo record is
  at 0/0; shutdown TRUE
  2004-01-05 08:49:10 LOG:  next transaction id: 88539506; next oid:
  88761828
  2004-01-05 08:49:10 LOG:  database system was not properly shut down;
  automatic recovery in progress
  2004-01-05 08:49:10 LOG:  ReadRecord: unexpected pageaddr 15/E640E000 in
  log file 21, segment 239, offset 4251648
  2004-01-05 08:49:10 LOG:  redo is not required
  2004-01-05 08:49:12 PANIC:  XLogWrite: write request 15/EF40E000 is past
  end of log 15/EF40E000
  2004-01-05 08:49:12 LOG:  startup process (pid 2907) was terminated by
  signal 6
  2004-01-05 08:49:12 LOG:  aborting startup due to startup process failure
  #============================================================================================

  Here is the ls -al from the pg_clog directory.
  I can't see any problems in log file 21:

  -rw-------    1 postgres postgres   262144 Jul  6  2003 0000
  -rw-------    1 postgres postgres   262144 Jul 10 02:10 0001
  -rw-------    1 postgres postgres   262144 Jul 16 00:24 0002
  -rw-------    1 postgres postgres   262144 Jul 21 21:15 0003
  -rw-------    1 postgres postgres   262144 Jul 27 11:53 0004
  -rw-------    1 postgres postgres   262144 Jul 29 11:11 0005
  -rw-------    1 postgres postgres   262144 Jul 30 18:41 0006
  -rw-------    1 postgres postgres   262144 Aug  3 20:24 0007
  -rw-------    1 postgres postgres   262144 Aug  5 02:07 0008
  -rw-------    1 postgres postgres   262144 Aug  6 18:52 0009
  -rw-------    1 postgres postgres   262144 Aug 11 17:51 000A
  -rw-------    1 postgres postgres   262144 Aug 12 18:50 000B
  -rw-------    1 postgres postgres   262144 Aug 14 02:34 000C
  -rw-------    1 postgres postgres   262144 Aug 15 13:40 000D
  -rw-------    1 postgres postgres   262144 Aug 17 18:04 000E
  -rw-------    1 postgres postgres   262144 Aug 20 15:58 000F
  -rw-------    1 postgres postgres   262144 Aug 24 10:09 0010
  -rw-------    1 postgres postgres   262144 Aug 25 14:52 0011
  -rw-------    1 postgres postgres   262144 Aug 27 15:15 0012
  -rw-------    1 postgres postgres   262144 Aug 31 14:58 0013
  -rw-------    1 postgres postgres   262144 Sep  8 10:05 0014
  -rw-------    1 postgres postgres   262144 Sep  8 13:55 0015
  -rw-------    1 postgres postgres   262144 Sep  8 23:46 0016
  -rw-------    1 postgres postgres   262144 Sep  9 21:01 0017
  -rw-------    1 postgres postgres   262144 Sep 10 02:00 0018
  -rw-------    1 postgres postgres   262144 Sep 10 09:49 0019
  -rw-------    1 postgres postgres   262144 Sep 10 12:16 001A
  -rw-------    1 postgres postgres   262144 Sep 10 16:28 001B
  -rw-------    1 postgres postgres   262144 Sep 11 04:51 001C
  -rw-------    1 postgres postgres   262144 Sep 11 14:00 001D
  -rw-------    1 postgres postgres   262144 Sep 14 13:57 001E
  -rw-------    1 postgres postgres   262144 Sep 15 02:02 001F
  -rw-------    1 postgres postgres   262144 Sep 15 10:41 0020
  -rw-------    1 postgres postgres   262144 Sep 15 12:37 0021
  -rw-------    1 postgres postgres   262144 Sep 15 15:19 0022
  -rw-------    1 postgres postgres   262144 Sep 15 22:58 0023
  -rw-------    1 postgres postgres   262144 Sep 16 13:26 0024
  -rw-------    1 postgres postgres   262144 Sep 17 06:43 0025
  -rw-------    1 postgres postgres   262144 Sep 17 15:27 0026
  -rw-------    1 postgres postgres   262144 Sep 18 13:14 0027
  -rw-------    1 postgres postgres   262144 Sep 21 13:37 0028
  -rw-------    1 postgres postgres   262144 Sep 22 09:21 0029
  -rw-------    1 postgres postgres   262144 Sep 22 15:44 002A
  -rw-------    1 postgres postgres   262144 Sep 23 01:15 002B
  -rw-------    1 postgres postgres   262144 Sep 23 10:49 002C
  -rw-------    1 postgres postgres   262144 Sep 23 14:36 002D
  -rw-------    1 postgres postgres   262144 Sep 24 00:32 002E
  -rw-------    1 postgres postgres   262144 Sep 24 11:56 002F
  -rw-------    1 postgres postgres   262144 Sep 24 22:18 0030
  -rw-------    1 postgres postgres   262144 Sep 25 14:11 0031
  -rw-------    1 postgres postgres   262144 Sep 29 12:43 0032
  -rw-------    1 postgres postgres   262144 Sep 30 00:31 0033
  -rw-------    1 postgres postgres   262144 Sep 30 18:49 0034
  -rw-------    1 postgres postgres   262144 Oct  2 14:04 0035
  -rw-------    1 postgres postgres   262144 Oct 19 08:42 0036
  -rw-------    1 postgres postgres   262144 Oct 19 12:59 0037
  -rw-------    1 postgres postgres   262144 Oct 19 16:11 0038
  -rw-------    1 postgres postgres   262144 Oct 20 10:18 0039
  -rw-------    1 postgres postgres   262144 Oct 20 14:40 003A
  -rw-------    1 postgres postgres   262144 Oct 21 09:03 003B
  -rw-------    1 postgres postgres   262144 Oct 21 15:06 003C
  -rw-------    1 postgres postgres   262144 Oct 22 11:47 003D
  -rw-------    1 postgres postgres   262144 Oct 22 23:09 003E
  -rw-------    1 postgres postgres   262144 Oct 23 16:15 003F
  -rw-------    1 postgres postgres   262144 Oct 26 11:09 0040
  -rw-------    1 postgres postgres   262144 Oct 26 15:16 0041
  -rw-------    1 postgres postgres   262144 Oct 27 11:00 0042
  -rw-------    1 postgres postgres   262144 Oct 27 15:42 0043
  -rw-------    1 postgres postgres   262144 Oct 28 10:45 0044
  -rw-------    1 postgres postgres   262144 Oct 28 19:55 0045
  -rw-------    1 postgres postgres   262144 Oct 29 13:31 0046
  -rw-------    1 postgres postgres   262144 Oct 30 11:04 0047
  -rw-------    1 postgres postgres   262144 Oct 31 15:34 0048
  -rw-------    1 postgres postgres   262144 Nov  2 15:27 0049
  -rw-------    1 postgres postgres   262144 Nov  3 16:16 004A
  -rw-------    1 postgres postgres   262144 Nov  4 18:06 004B
  -rw-------    1 postgres postgres   262144 Nov  6 08:59 004C
  -rw-------    1 postgres postgres   262144 Nov  9 10:24 004D
  -rw-------    1 postgres postgres   262144 Nov 10 12:55 004E
  -rw-------    1 postgres postgres   262144 Nov 11 16:06 004F
  -rw-------    1 postgres postgres   262144 Nov 12 19:03 0050
  -rw-------    1 postgres postgres   262144 Nov 13 15:45 0051
  -rw-------    1 postgres postgres   262144 Nov 20 12:14 0052
  -rw-------    1 postgres postgres   262144 Dec 17 19:56 0053
  -rw-------    1 postgres postgres   122880 Jan  5 00:08 0054





Re: postgresql crushed with XLogWrite error

From
Andrew Sullivan
Date:
On Mon, Jan 05, 2004 at 02:14:37PM +0200, Tsirkin Evgeny wrote:
>  Hello dear list!
>  I am using postgresql 7.3.3 .Last night the errors occured.
                             ^

>  by now ,but what happened?Is that a known bug?What to do next time?

Yes, it's a known bug, and is the very reason 7.3.4 was released.
Upgrade.

A

--
----
Andrew Sullivan                         204-4141 Yonge Street
Afilias Canada                        Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Re: postgresql crushed with XLogWrite error

From
Tsirkin Evgeny
Date:
Thanks!
Now another two questions:
Is there any official bug report on this that i can show to
my boss?
Should i upgrade to 7.3.4 to fix that bug or it is better to get newly
version?
What i need is to restore my corrupted db from the tape ,upgrade postgresql
(say from rpm) and just start the server.I don't want to do all the dump -
restore
thing.

On Mon, 5 Jan 2004 08:49:13 -0500, Andrew Sullivan
<andrew@libertyrms.info> wrote:

> On Mon, Jan 05, 2004 at 02:14:37PM +0200, Tsirkin Evgeny wrote:
>>  Hello dear list!
>>  I am using postgresql 7.3.3 .Last night the errors occured.
>                              ^
>
>>  by now ,but what happened?Is that a known bug?What to do next time?
>
> Yes, it's a known bug, and is the very reason 7.3.4 was released.
> Upgrade.
>
> A
>




Re: postgresql crushed with XLogWrite error

From
Andrew Sullivan
Date:
On Mon, Jan 05, 2004 at 04:07:34PM +0200, Tsirkin Evgeny wrote:
> Thanks!
> Now another two questions:
> Is there any official bug report on this that i can show to
> my boss?

Yes, it's in the release notes.  IIRC, there were like 24 or 48 hours
between the releases because of this.  Tom Lane occasionally beats
himself up about the bug on-list.

> Should i upgrade to 7.3.4 to fix that bug or it is better to get newly
> version?

You can upgrade to the lastest 7.3.x release by just dropping the new
binary in place, IIRC -- read the release notes.  In any case, you'll
need to get some version of 7.3 running first.

A

----
Andrew Sullivan                         204-4141 Yonge Street
Afilias Canada                        Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Re: postgresql crushed with XLogWrite error

From
Tsirkin Evgeny
Date:
Probably a dumm question ,but it is better to ask then be sorry.
Nothing is changed in config files ,so i am extractiing
the binary from rpm ,copy it to the place and start the server,right?
And thanks again!
On Mon, 5 Jan 2004 09:09:55 -0500, Andrew Sullivan
<andrew@libertyrms.info> wrote:

> On Mon, Jan 05, 2004 at 04:07:34PM +0200, Tsirkin Evgeny wrote:
>> Thanks!
>> Now another two questions:
>> Is there any official bug report on this that i can show to
>> my boss?
>
> Yes, it's in the release notes.  IIRC, there were like 24 or 48 hours
> between the releases because of this.  Tom Lane occasionally beats
> himself up about the bug on-list.
>
>> Should i upgrade to 7.3.4 to fix that bug or it is better to get newly
>> version?
>
> You can upgrade to the lastest 7.3.x release by just dropping the new
> binary in place, IIRC -- read the release notes.  In any case, you'll
> need to get some version of 7.3 running first.
>
> A
>
> ----
> Andrew Sullivan                         204-4141 Yonge Street
> Afilias Canada                        Toronto, Ontario Canada
> <andrew@libertyrms.info>                              M2P 2A8
>                                          +1 416 646 3304 x110
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster




Re: postgresql crushed with XLogWrite error

From
Andrew Sullivan
Date:
On Mon, Jan 05, 2004 at 04:48:00PM +0200, Tsirkin Evgeny wrote:
> Probably a dumm question ,but it is better to ask then be sorry.
> Nothing is changed in config files ,so i am extractiing
> the binary from rpm ,copy it to the place and start the server,right?
> And thanks again!

If you're upgrading via RPM, why not just do a regular RPM
installation?  Doesn't that work?  (Extracting binaries from RPMs and
doing other such things sounds like the way to break something.)

A

--
----
Andrew Sullivan                         204-4141 Yonge Street
Afilias Canada                        Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Re: postgresql crushed with XLogWrite error

From
Tsirkin Evgeny
Date:
On Mon, 5 Jan 2004 11:23:08 -0500, Andrew Sullivan
<andrew@libertyrms.info> wrote:

I want to keep my configs.Of course i can save my postgresql.conf
(and other files that i changed and don't currently remember)
to another location and then put it back in place of what rpm will
put ,but as i said ,i don't actually remember what i have changed :(
>
> If you're upgrading via RPM, why not just do a regular RPM
> installation?  Doesn't that work?  (Extracting binaries from RPMs and
> doing other such things sounds like the way to break something.)
>
> A
>




Re: postgresql crushed with XLogWrite error

From
Andrew Sullivan
Date:
On Mon, Jan 05, 2004 at 07:35:16PM +0200, Tsirkin Evgeny wrote:
> On Mon, 5 Jan 2004 11:23:08 -0500, Andrew Sullivan
> <andrew@libertyrms.info> wrote:
>
> I want to keep my configs.Of course i can save my postgresql.conf
> (and other files that i changed and don't currently remember)
> to another location and then put it back in place of what rpm will
> put ,but as i said ,i don't actually remember what i have changed :(

The only things that should make a difference to you should be in the
/etc directory, assuming the RPM follows the filesystem standard (I
believe it does).  My suggestion is to back everything up, then
replace the RPM in the standard way, and then compare the files one
at a time to see what's different.  I'll bet the binaries and
libraries, of course, and things under /etc/[postgres?]/, and that's
it.  Most likely, it'll be postgresql.conf and pg_hba.conf, and maybe
pg_ident.conf.

A

--
----
Andrew Sullivan                         204-4141 Yonge Street
Afilias Canada                        Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Re: postgresql crushed with XLogWrite error

From
Tsirkin Evgeny
Date:
On Mon, 5 Jan 2004 12:39:20 -0500, Andrew Sullivan
<andrew@libertyrms.info> wrote:

>
> The only things that should make a difference to you should be in the
> /etc directory, assuming the RPM follows the filesystem standard (I
> believe it does).
It does not ,rpm puts the staff in /var/lib

> My suggestion is to back everything up, then
> replace the RPM in the standard way, and then compare the files one
> at a time to see what's different.  I'll bet the binaries and
> libraries, of course, and things under /etc/[postgres?]/, and that's
> it.  Most likely, it'll be postgresql.conf and pg_hba.conf, and maybe
> pg_ident.conf.
>
> A
>
Well, this is what i eventually did ,it was just more convinient to
wget the rpms ,type rpm -Uhv postgres*rpm and then replace 2 files.
(It took some work to recall what i have changed in the config
files,thought)
That was simpler then rpm2cpio etc... as i thought at the beginning.
The server works fine by now,hope this XLogWrite thing will not come
back .
Thanks for the help.
Evgeny




Re: postgresql crushed with XLogWrite error

From
Christopher Browne
Date:
andrew@libertyrms.info (Andrew Sullivan) writes:
> On Mon, Jan 05, 2004 at 07:35:16PM +0200, Tsirkin Evgeny wrote:
>> On Mon, 5 Jan 2004 11:23:08 -0500, Andrew Sullivan
>> <andrew@libertyrms.info> wrote:
>>
>> I want to keep my configs.Of course i can save my postgresql.conf
>> (and other files that i changed and don't currently remember)
>> to another location and then put it back in place of what rpm will
>> put ,but as i said ,i don't actually remember what i have changed :(
>
> The only things that should make a difference to you should be in the
> /etc directory, assuming the RPM follows the filesystem standard (I
> believe it does).  My suggestion is to back everything up, then
> replace the RPM in the standard way, and then compare the files one
> at a time to see what's different.  I'll bet the binaries and
> libraries, of course, and things under /etc/[postgres?]/, and that's
> it.  Most likely, it'll be postgresql.conf and pg_hba.conf, and maybe
> pg_ident.conf.

It's easy enough to check what files the RPM intends to update...

bash-2.05a$ rpm -q -p -l netpbm-devel-9.24-3.i386.rpm
/usr/include/pam.h
/usr/include/pammap.h
/usr/include/pbm.h
/usr/include/pbmshhopt.h
/usr/include/pgm.h
/usr/include/pm.h
... other files omitted ...

That can diminish how much is in the "everything" that needs to be
backed up...
--
let name="cbbrowne" and tld="libertyrms.info" in String.concat "@" [name;tld];;
<http://dev6.int.libertyrms.com/>
Christopher Browne
(416) 646 3304 x124 (land)