Thread: pg_clog woes with 7.3.2 - Episode 2
Hi all, A week or 2 back I reported a problem with 7.3.2 failing to open pg_clog files ([HACKERS] pg_clog woes with 7.3.2). Tom Lane was kind enough to login and do some debugging on a couple of occasions and found corrupt pages in a database on the system. After the first session, he suggested running memtest86 which showed no errors after multiple passes and badblocks which showed no errors after multiple non-destructive read-write tests. One initdb and reload later (it's a new system, the old is still running OK), and the error comes back again, only this time Tom finds the corruption is in a couple of pages. Memtest86 again shows no errors, but eventually badblocks did, but only when I used a destructive read write test. So, the disk goes back to Seagate, and is replaced with another identical one, and a similar problem reoccurs (logs below) :-(. I haven't run badblocks yet as it takes a fair while, but wanted to find out if anyone thought this could be an OS issue or something else. Previously I've been using the 2.4.19 Linux kernel, however this machine is 2.4.20 (Slackware Linux 9). The SCSI adaptor is an Adaptec 29160, and the disks are 34Gb Seagate Cheetah X15's. Any thoughts or suggestions would be appreciated. Regards, Dave. LOG: connection received: host=[local] LOG: connection authorized: user=postgres database=mnogo_int LOG: query: begin; select getdatabaseencoding(); commit LOG: query: vacuum; PANIC: read of clog file 0, offset 253952 failed: Success LOG: statement: vacuum; LOG: server process (pid 2006) was terminated by signal 6 LOG: terminating any other active server processes LOG: all server processes terminated; reinitializing shared memory and semaphores LOG: database system was interrupted at 2003-04-16 15:06:34 BST LOG: checkpoint record is at 0/2FB37D94 LOG: redo record is at 0/2FB37D94; undo record is at 0/0; shutdown TRUE LOG: next transaction id: 3186; next oid: 9724496 LOG: database system was not properly shut down; automatic recovery in progress LOG: redo starts at 0/2FB37DD4 LOG: ReadRecord: record with zero length at 0/2FC39CAC LOG: redo done at 0/2FC39C88 LOG: database system is ready
"Dave Page" <dpage@vale-housing.co.uk> writes:
> One initdb and reload later (it's a new system, the old is still running
> OK), and the error comes back again, only this time Tom finds the
> corruption is in a couple of pages. Memtest86 again shows no errors, but
> eventually badblocks did, but only when I used a destructive read write
> test.
> So, the disk goes back to Seagate, and is replaced with another
> identical one, and a similar problem reoccurs (logs below) :-(. I
> haven't run badblocks yet as it takes a fair while, but wanted to find
> out if anyone thought this could be an OS issue or something else.
> Previously I've been using the 2.4.19 Linux kernel, however this machine
> is 2.4.20 (Slackware Linux 9). The SCSI adaptor is an Adaptec 29160, and
> the disks are 34Gb Seagate Cheetah X15's.
How annoying. My bet would be on the SCSI adaptor being the problem.
Or you could have a cabling issue --- SCSI is not as bad as IDE, but
it's still finicky, esp w.r.t. termination.
regards, tom lane
Dave Page wrote: > So, the disk goes back to Seagate, and is replaced with another > identical one, and a similar problem reoccurs (logs below) :-(. I > haven't run badblocks yet as it takes a fair while, but wanted to find > out if anyone thought this could be an OS issue or something else. > Previously I've been using the 2.4.19 Linux kernel, however this machine > is 2.4.20 (Slackware Linux 9). The SCSI adaptor is an Adaptec 29160, and > the disks are 34Gb Seagate Cheetah X15's. > > Any thoughts or suggestions would be appreciated. I'd definitely run badblocks against the new drive -- multiple times. Either it should yield the same bad block list each time (in which case you've got a set of unrecoverable bad block -- this usually means there are no spare blocks left), or you should see the number of bad blocks drop to zero (as the SCSI bad block remapping takes effect), unless something really funky is going on. I'd also start looking carefully through the system logs for SCSI errors. You should see some if you're getting bad block problems (in particular, you should see bad block remapping attempts that couldn't read the data from the original bad block -- this, or running out of spare blocks, is the only reason you should see errors at all on an otherwise functional setup). If badblocks shows errors but you don't see any SCSI errors in the system logs, then it's time to start suspecting the disk controller or perhaps even the PCI bus controller, because it means something really weird is happening on the backend that is entirely invisible. Cabling or termination could be an issue, but I'd expect to see parity errors, timed out commands, etc. if that's the problem. -- Kevin Brown kevin@sysexperts.com
Kevin Brown <kevin@sysexperts.com> writes:
> If badblocks shows errors but you don't see any SCSI errors in the
> system logs, then it's time to start suspecting the disk controller or
> perhaps even the PCI bus controller, because it means something really
> weird is happening on the backend that is entirely invisible. Cabling
> or termination could be an issue, but I'd expect to see parity errors,
> timed out commands, etc. if that's the problem.
Dave neglected to mention that the two or three bad blocks we'd traced
down all showed a consistent pattern of errors: there was a 64-byte
region of wrong data, aligned on a 64-byte offset from the start of the
disk block, and the contents were copies of correct data from positions
exactly 64 bytes before or after the bad area.
Considering that, I would bet a good deal that the problem is some kind
of transfer timing error in some chunk of hardware that copies the data
64 bytes at a time. I withdraw my previous thought that it might be
cabling --- there are no 64-byte-wide SCSI cables. It could easy be
internal to the SCSI adaptor though. If his motherboard is high-end
enough that the DMA path from adaptor to memory is 64 bytes wide, then
DMA timing errors would be a possibility too.
regards, tom lane
> -----Original Message----- > From: Tom Lane [mailto:tgl@sss.pgh.pa.us] > Sent: 16 April 2003 16:22 > To: Dave Page > Cc: pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] pg_clog woes with 7.3.2 - Episode 2 > > > How annoying. My bet would be on the SCSI adaptor being the > problem. Or you could have a cabling issue --- SCSI is not as > bad as IDE, but it's still finicky, esp w.r.t. termination. OK, I've now tried a 29160N adaptor with a DPT cable (with non-removable terminator as per the previous Adaptec cable), and have even removed the DAT drive from the system and I still see the same sort of errors following an initdb/reload. A couple of things I've noticed: - It always crashes on the same database which is some way through the original dump file and my vacuum script. If I drop that one it may or may not crash on another one further through the script. - Sometimes everything works fine until I reboot the system. I'll see if I can downgrade the kernel later today though it's all a bit hectic with the easter break tomorrow so it may have to wait until next week. Still, if anyone has any more ideas, I'd be grateful... Regards, Dave.
Hi Kevin > -----Original Message----- > From: Kevin Brown [mailto:kevin@sysexperts.com] > Sent: 17 April 2003 03:35 > To: pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] pg_clog woes with 7.3.2 - Episode 2 > > I'd also start looking carefully through the system logs for > SCSI errors. You should see some if you're getting bad block > problems (in particular, you should see bad block remapping > attempts that couldn't read the data from the original bad > block -- this, or running out of spare blocks, is the only > reason you should see errors at all on an otherwise functional setup). No errors on the system at all apart from PostgreSQL. On the previous disk, the number of errors reported by badblocks was rising from 64 initially to 80-something when I took it out. > If badblocks shows errors but you don't see any SCSI errors > in the system logs, then it's time to start suspecting the > disk controller or perhaps even the PCI bus controller, > because it means something really weird is happening on the > backend that is entirely invisible. Cabling or termination > could be an issue, but I'd expect to see parity errors, timed > out commands, etc. if that's the problem. Yes, me too. Still, I've now tried a 29160N adaptor, and changed the cable to a DPT one (both that and the previous Adaptec have hardwired terminators). I've also removed the DAT drive from the system so the only things on the SCSI bus are 2 identical disks and the adaptor. I'm beginning to wonder about my shiny new 2.4.20 kernel... Regards, Dave.
> -----Original Message----- > From: Tom Lane [mailto:tgl@sss.pgh.pa.us] > Sent: 17 April 2003 04:40 > To: Kevin Brown > Cc: pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] pg_clog woes with 7.3.2 - Episode 2 > > > Considering that, I would bet a good deal that the problem is > some kind of transfer timing error in some chunk of hardware > that copies the data 64 bytes at a time. I withdraw my > previous thought that it might be cabling --- there are no > 64-byte-wide SCSI cables. It could easy be internal to the > SCSI adaptor though. I've now tried a 29160 and a 29160N both exhibiting the problem. I also have a 29160 in the existing server I'm trying to replace and that works fine. > If his motherboard is high-end enough > that the DMA path from adaptor to memory is 64 bytes wide, > then DMA timing errors would be a possibility too. This is a relatively low-end Gigbyte dual PIII board - GA6-VTXD rev 1.0. Again, I have more of these that are working fine (though aren't running PostgreSQL). Do you think it's worth trying the older kernel that I know works? Could a bug in 2.4.20 give this sort of error? Regards, Dave.
"Dave Page" <dpage@vale-housing.co.uk> writes:
> Do you think it's worth trying the older kernel that I know works? Could
> a bug in 2.4.20 give this sort of error?
It still smells like a hardware issue to me --- but I'm running out of
ideas ... You might as well try the other kernel to see what happens.
regards, tom lane
Dave Page wrote: > Still, I've now tried a 29160N adaptor, and changed the > cable to a DPT one (both that and the previous Adaptec have hardwired > terminators). I've also removed the DAT drive from the system so the > only things on the SCSI bus are 2 identical disks and the adaptor. > > I'm beginning to wonder about my shiny new 2.4.20 kernel... That's a possibility. There's one other I'd try, prior to using a different kernel: try booting the kernel with the "noapic" option. On certain SMP systems, this has allowed the kernel to come up properly and see the SCSI adaptor (among other things), whereas without it the system would hang on attempts to access the SCSI device. Even if you're not running SMP, it may help your system's stability. -- Kevin Brown kevin@sysexperts.com
Kevin Brown wrote: > Dave Page wrote: > > Still, I've now tried a 29160N adaptor, and changed the > > cable to a DPT one (both that and the previous Adaptec have hardwired > > terminators). I've also removed the DAT drive from the system so the > > only things on the SCSI bus are 2 identical disks and the adaptor. > > > > I'm beginning to wonder about my shiny new 2.4.20 kernel... > > That's a possibility. There's one other I'd try, prior to using a > different kernel: try booting the kernel with the "noapic" option. On > certain SMP systems, this has allowed the kernel to come up properly > and see the SCSI adaptor (among other things), whereas without it the > system would hang on attempts to access the SCSI device. Even if > you're not running SMP, it may help your system's stability. The "noapic" option seems a quasi-magical elixir for many sorts of ailments. I upgraded a box to 2.4.20 and discovered that my NIC was no longer properly recognized until I threw that option in. Others in my office have /apparently/ the same hardware, and found they didn't need the option. As a "fix," it certainly seems to fall into the "snakeoil/superstition" category. While it often seems to have a useful effect, I haven't located any actual explanations as to why it should be expected to work. But based on the comments in the thread, I would concur that my suspicions would be with the kernel as the most likely root of the problem. -- (reverse (concatenate 'string "moc.enworbbc@" "enworbbc")) http://www3.sympatico.ca/cbbrowne/nonrdbms.html Rules of the Evil Overlord #82. "I will not shoot at any of my enemies if they are standing in front of the crucial support beam to a heavy, dangerous, unbalanced structure. <http://www.eviloverlord.com/>
Thanks Kevin, I will try that when I return to the office on Wednesday. Regards, Dave. > -----Original Message----- > From: Kevin Brown [mailto:kevin@sysexperts.com] > Sent: 18 April 2003 10:58 > To: pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] pg_clog woes with 7.3.2 - Episode 2 > > > Dave Page wrote: > > Still, I've now tried a 29160N adaptor, and changed the > > cable to a DPT one (both that and the previous Adaptec have > hardwired > > terminators). I've also removed the DAT drive from the > system so the > > only things on the SCSI bus are 2 identical disks and the adaptor. > > > > I'm beginning to wonder about my shiny new 2.4.20 kernel... > > That's a possibility. There's one other I'd try, prior to > using a different kernel: try booting the kernel with the > "noapic" option. On certain SMP systems, this has allowed > the kernel to come up properly and see the SCSI adaptor > (among other things), whereas without it the system would > hang on attempts to access the SCSI device. Even if you're > not running SMP, it may help your system's stability. > > > > > -- > Kevin Brown kevin@sysexperts.com > > > ---------------------------(end of > broadcast)--------------------------- > TIP 4: Don't 'kill -9' the postmaster >
cbbrowne@cbbrowne.com wrote: > The "noapic" option seems a quasi-magical elixir for many sorts of > ailments. > > I upgraded a box to 2.4.20 and discovered that my NIC was no longer > properly recognized until I threw that option in. Others in my > office have /apparently/ the same hardware, and found they didn't > need the option. > > As a "fix," it certainly seems to fall into the > "snakeoil/superstition" category. While it often seems to have a > useful effect, I haven't located any actual explanations as to why > it should be expected to work. Well, when it comes to booting a computer, the placebo effect doesn't really exist. :-) Normally I'd agree that "noapic" sounds and smells like snakeoil. The problem is that it has observable and repeatable effects on some systems, and thus can't really be classified as snakeoil (much as one might like to!). Why should it be expected to work? I don't know...possibly because the APIC hardware is buggy (perhaps in very subtle ways) on some systems? Possibly because the APIC driver is subtlely incompatible with certain APIC hardware? Possibly because the APIC driver has certain subtle bugs that only manifest themselves on certain motherboards with certain peripheral devices? Whatever the reason, the "noapic" option *does* work on certain systems, so it unfortunately isn't something that can be dismissed as mere superstition -- the computer isn't being asked its opinion of its own health here, nor does it "know" that it should get "well" when given different boot options. No "placebo effect" involved, just repeatable observation (that the observation isn't terribly repeatable *across* systems does not diminish the validity of the observation). -- Kevin Brown kevin@sysexperts.com
On Fri, 18 Apr 2003, Kevin Brown wrote: > cbbrowne@cbbrowne.com wrote: > > The "noapic" option seems a quasi-magical elixir for many sorts of > > ailments. > > > > I upgraded a box to 2.4.20 and discovered that my NIC was no longer > > properly recognized until I threw that option in. Others in my > > office have /apparently/ the same hardware, and found they didn't > > need the option. > > > > As a "fix," it certainly seems to fall into the > > "snakeoil/superstition" category. While it often seems to have a > > useful effect, I haven't located any actual explanations as to why > > it should be expected to work. > > Well, when it comes to booting a computer, the placebo effect doesn't > really exist. :-) > > Normally I'd agree that "noapic" sounds and smells like snakeoil. The > problem is that it has observable and repeatable effects on some > systems, and thus can't really be classified as snakeoil (much as one > might like to!). > > Why should it be expected to work? I don't know...possibly because > the APIC hardware is buggy (perhaps in very subtle ways) on some > systems? Possibly because the APIC driver is subtlely incompatible > with certain APIC hardware? Possibly because the APIC driver has > certain subtle bugs that only manifest themselves on certain > motherboards with certain peripheral devices? > > Whatever the reason, the "noapic" option *does* work on certain > systems, so it unfortunately isn't something that can be dismissed as > mere superstition -- the computer isn't being asked its opinion of its > own health here, nor does it "know" that it should get "well" when > given different boot options. No "placebo effect" involved, just > repeatable observation (that the observation isn't terribly repeatable > *across* systems does not diminish the validity of the observation). Just to add to this, on some of the first SMP systems I messed with there was a setting for some SMP version of 1.1 or 1.4, and using 1.1 resulted in an unstable box for me. 1.4 fixed all the issues. SMP on Intel is a wild ride, and no two motherboards are equivalent. I've had good luck with Supermicro and Intel SMP motherboards, although both have needed BIOS updates at times.
Keven Brown wrote: > Normally I'd agree that "noapic" sounds and smells like snakeoil. The > problem is that it has observable and repeatable effects on some > systems, and thus can't really be classified as snakeoil (much as one > might like to!). There's a /slight/ difference between 'superstition' and 'snakeoil'; the latter is something you don't expect to find effectual. The former may represent something you don't/can't understand. Recall Clarke's observation that "Any sufficiently advanced technology is indistinguishable from magic." Anything I can't explain is likely to either be: a) Something I don't understand yet, or b) Perhaps something truly supernatural, that cannot be explained based on any sort of natural reasoning. Perhaps explainableby "God did something unexplainable, therefore things are the way they are." There are disagreements as to where to draw the line. People of some degrees of "superstitiousness" may be prepared to explain _everything_ as involving "God decided to make that happen," making _no_ attempt to understand lower level processes. Some, more skeptical, may reject that anything should be able to fall into category b). On the other hand, some people get themselves rip-roaring drunk and trip over the line we thought was one of concept: "SCSI is *NOT* magic. There are *fundamental technical reasons* why it is necessary to sacrifice a young goat to yourSCSI chain now and then." -- If this was helpful, <http://svcs.affero.net/rm.php?r=cbbrowne> rate me http://www3.sympatico.ca/cbbrowne/wp.html "While preceding your entrance with a grenade is a good tactic in Quake, it can lead to problems if attempted at work." -- C Hacking -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html
Just caught up on this thread - I currently have the same problem on a Duron 900 with a Gigabyte 7VAX (or similar) motherboard. We've had a number of problems with this system before, but to date they were all fixed by increasing the available shared memory (echo $BIGNUM > /proc/sys/kernel/shmall). We're on a bog standard IDE drive. I've not yet run so many tests, but will advise of results shortly. -- Posted via http://dbforums.com