Thread: Re: Strange database corruption with PostgreSQL 7.4.x o

Hello Scott!

Thank you for the quick answer. I'll try to check our hardware which is a
Compaq DL380 G4 with a batteyr buffered write cache on our raid controller.
As the system is running stable at all i think it's not the cpu or memory.
At moment i tend more to a bad disk or SCSI controller but even with that i
don't get any message in my logs...
Any ideas how i could check the hardware?

Best regards,
Matthias

> -----Original Message-----
> From: pgsql-general-owner@postgresql.org
> [mailto:pgsql-general-owner@postgresql.org] On Behalf Of Scott Marlowe
> Sent: Wednesday, September 20, 2006 2:56 PM
> To: Matthias.Pitzl@izb.de
> Cc: pgsql-general@postgresql.org
> Subject: Re: [GENERAL] Strange database corruption with
> PostgreSQL 7.4.x on
>
>
> On Wed, 2006-09-20 at 14:34 +0200, Matthias.Pitzl@izb.de wrote:
> > Hello!
> >
> > We're running the latest release of PostgreSQL 7.4.13 on a
> Debian Sarge
> > machine. Postgres has been compiled by oureselves.
> > We have a pretty big database running on this machine, it
> has about 6.4 GB
> > approximately. One table contains about 55 million rows.
> > Into this table we insert about 500000 rows each day. Our
> problem is that
> > without any obvious reason the database gets corrupt. The
> messages we get
> > are:
> > invalid page header in block 437702 of relation "xxxx"
> > We already have tried out some other versions of 7.4. On
> another machine
> > running Debian Woody with PotgreSQL 7.4.10 we don't have
> any problems.
> > Kernels are 2.4.33 on the Sarge machine, 2.4.28 on the
> Woody machine. Both
> > are SMP kernels.
> > Does anyone of you perhaps have some hints what's going wrong here?
>
> Most likely causes in these cases tends to be, bad memory, bad hard
> drive, bad cpu, bad RAID / IDE / SCSI controller, loss of power when
> writing to IDE drives / RAID controllers with cache with no battery
> backup.
>
> I.e. check your hardware.
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 6: explain analyze is your friend
>

On Wed, 2006-09-20 at 15:14 +0200, Matthias.Pitzl@izb.de wrote:
> Hello Scott!
>
> Thank you for the quick answer. I'll try to check our hardware which is a
> Compaq DL380 G4 with a batteyr buffered write cache on our raid controller.
> As the system is running stable at all i think it's not the cpu or memory.
> At moment i tend more to a bad disk or SCSI controller but even with that i
> don't get any message in my logs...
> Any ideas how i could check the hardware?

Keep in mind, a single bad memory location is all it takes to cause data
corruption, so it could well be memory.  CPU is less likely if the
machine is otherwise running stable.

The standard tool on x86 hardware is memtest86 www.memtest86.com

So, you'd have to schedule a maintenance window to run the test in since
you have to basically down the machine and run just memtest86.  I think
a few live linux distros have it built in (FC has a memtest label in
some versions I think)

My first suspicion is always memory.  We ordered a batch of memory from
a very off brand supplier, and over 75% tested bad.  And it took >24
hours to find some of the bad memory.

good luck with your testing, let us know how it goes.

On Wed, 20 Sep 2006, Matthias.Pitzl@izb.de wrote:

> Any ideas how i could check the hardware?

1. memtest86 or memtest86+ at least 8 hours

2. CPU Burn-in
http://users.bigpond.net.au/cpuburn/ at least 8 hours

3. badblocks -s -v -t random /dev/sd%
WARNING: this will destroy your data!

4. smartctl -a /dev/sd%
Does not have to work. Sometimes needs some hacking to make it work.

Dave "KernelSlacker" Jones wrote a very good article about hardware
problems:
http://people.redhat.com/davej/hardware-problems.txt

Regards
Tometzky
--
...although Eating Honey was a very good thing to do, there was a
moment just before you began to eat it which was better than when you
were...
                                                      Winnie the Pooh