Re: 9.3: summary of corruption detection / checksums / CRCs discussion - Mailing list pgsql-hackers

From Merlin Moncure
Subject Re: 9.3: summary of corruption detection / checksums / CRCs discussion
Date
Msg-id CAHyXU0w=cYTjYJ3NZB9BD76NaNtKUMF6kbEQ3Z+LuvoZCxWDGg@mail.gmail.com
Whole thread Raw
In response to Re: 9.3: summary of corruption detection / checksums / CRCs discussion  (Merlin Moncure <mmoncure@gmail.com>)
List pgsql-hackers
On Wed, Apr 25, 2012 at 9:28 AM, Merlin Moncure <mmoncure@gmail.com> wrote:
>> That patch actually has more than one optimization in it, I think, but
>> the basic idea is that if we could figure out a way to set
>> HEAP_XMIN_COMMITTED when loading data into a table created or
>> truncated within the same transaction, the need to set hint bits on
>> first scan of the table would be eliminated.  Writing the xmin as
>> FrozenTransactionId would save even more, though it introduces some
>> additional complexity.
>
> This would be great but it's only a corner case.  A pretty common
> application flow is to write a large number of records, scan them,
> update them, scan them again, delete them, etc. in a table that's
> already established and possibly pretty large.  Unfortunately this
> type of work doesn't get a lot of coverage with the common benchmarks.
>
> Also, wouldn't the extra out of band wal traffic from hint bits
> exacerbate contention issues on the wal insert lock?

Hm,  You probably remember the process local hint bit cache.  While my
implementation had a lot of issues and it's arguable if I'm a patient
enough C programmer to come up with one to pass muster, it did work as
advertised: it greatly reduced to the point of elimination  hint bit
traffic in cases like this, including the bulk load case.  Some
variant of the approach would probably take at least some of the sting
out of the logged hint bits if that's the direction things have to go.

That doesn't help high concurrency cases where the #tuples/xid is low
though.  And even a small increase in contention on the wal insert
lock could mean a measurable reduction in scaling unless that problem
is solved independently.

merlin


pgsql-hackers by date:

Previous
From: Simon Riggs
Date:
Subject: Re: Temporary tables under hot standby
Next
From: Greg Stark
Date:
Subject: Re: patch submission: truncate trailing nulls from heap rows to reduce the size of the null bitmap