On Fri, Jan 27, 2012 at 1:31 PM, Bridget Frey <bridget.frey@redfin.com> wro=
te:
> Thanks for the info - that's very helpful.=A0 We had also noted that the =
alloc
> seems to be -3 bytes.=A0 We have run pg_check and it found no instances of
> corruption. We've also replayed queries that have failed, and have never
> been able to get the same query to fail twice.=A0 In the case you
> investigated, was there permanent page corruption - e.g. you could run the
> same query over and over and get the same result?
Yes. I observed that the infomask bits on several tuples had somehow
been overwritten by nonsense. I am not sure whether there were other
kinds of corruption as well - I suspect probably so - but that's the
only one I saw with my own eyes, courtesy of pg_filedump.
> It really does seem like this is an issue either in Hot Standby or very
> closely related to that feature, where there is temporary corruption of a
> btree index that then disappears.=A0 Our master is not experiencing any m=
alloc
> issues, while the 3 slaves get about a dozen per day, despite similar
> workloads.=A0 We haven't have a slave segfault since we set it up to prod=
uce a
> core dump, but we're expecting to have that within the next few days
> (assuming we'll continue to get a segfault every 3-4 days).=A0 We're also
> planning to set up one slave that will panic when it gets a malloc issue,=
as
> you (and other posters on 6400) had suggested.
>
> Thanks again for the help, and we'll keep you posted as we learn more...
The case I investigated involved corruption on the master, and I think
it predated Hot Standby. However, the symptom is generic enough that
it seems quite possible that there's more than one way for it to
happen. :-(
--=20
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
