We have no DDL whatsoever in the code. We do update rows in the
logins table frequently, but we basically have a policy of only doing
DDL changes during scheduled upgrades when we bring the site down. We
have been discussing this issue a lot and we really haven't come up
with anything that would be considered unusual here. The tables
experiencing issues have maybe 1M to 200M rows, we do updates and
selects frequently, they have standard btree primary key indexes, and
the failing query always seems to be a select for a single row based
on a primary key lookup. All of these code paths worked flawlessly
prior to our 9.1 upgrade (we had been using skytools replication).
And we see no problems on the master despite similar workloads there.
It is definitely puzzling, and we are not too sure what to look into
next...
Sent from my iPhone
On Jan 30, 2012, at 9:06 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I wrote:
>> Hm. The stack trace is definitive that it's finding the bad data in a
>> tuple that it's trying to print to the client, not in an index.
>
> BTW, after a bit more reflection it occurs to me that it's not so much
> that the data is necessarily *bad*, as that it seemingly doesn't match
> the tuple descriptor that the backend's trying to interpret it with.
> (In particular, the reported symptom would be consistent with finding
> a small integer constant at a place where the descriptor expects to find
> a variable-length field.) So that opens up a different line of thought
> about how those could get out of sync on a standby. Are you in the
> habit of issuing ALTER TABLE commands to add/delete/change columns on
> these tables? In fact, is there any DDL whatsoever going on around the
> time these failures happen?
>
> regards, tom lane
