Thread: old_snapshot_threshold's interaction with hash index

On Sun, May 1, 2016 at 1:43 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Sun, May 1, 2016 at 12:05 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> Currently we do the test for old snapshot (TestForOldSnapshot) for hash
>> indexes while scanning them.  Does this test makes any sense for hash
>> indexes considering LSN on hash index will always be zero (as hash indexes
>> are not WAL-logged)?  It seems to me that PageLSN check in
>> TestForOldSnapshot() will always return false which means that the error
>> "snapshot too old" won't be generated for hash indexes.
>>
>> Am I missing something here, if not, then I think we need a way to
>> prohibit pruning for hash indexes based on old_snapshot_threshold?
>
> What I mean to say here is prohibit pruning the relation which has hash
> index based on old_snapshot_threshold.

Good spot; added to the open issues page.

Thanks!

-- 
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Mon, May  2, 2016 at 04:02:35PM -0500, Kevin Grittner wrote:
> On Sun, May 1, 2016 at 1:43 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > On Sun, May 1, 2016 at 12:05 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >>
> >> Currently we do the test for old snapshot (TestForOldSnapshot) for hash
> >> indexes while scanning them.  Does this test makes any sense for hash
> >> indexes considering LSN on hash index will always be zero (as hash indexes
> >> are not WAL-logged)?  It seems to me that PageLSN check in
> >> TestForOldSnapshot() will always return false which means that the error
> >> "snapshot too old" won't be generated for hash indexes.
> >>
> >> Am I missing something here, if not, then I think we need a way to
> >> prohibit pruning for hash indexes based on old_snapshot_threshold?
> >
> > What I mean to say here is prohibit pruning the relation which has hash
> > index based on old_snapshot_threshold.
> 
> Good spot; added to the open issues page.

Uh, I have no idea how this would be fixed if the PageLSN is zero.  Do
you?

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

On Tue, May 3, 2016 at 10:45 AM, Bruce Momjian <bruce@momjian.us> wrote:
> On Mon, May  2, 2016 at 04:02:35PM -0500, Kevin Grittner wrote:
>> On Sun, May 1, 2016 at 1:43 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> > On Sun, May 1, 2016 at 12:05 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> >>
>> >> Currently we do the test for old snapshot (TestForOldSnapshot) for hash
>> >> indexes while scanning them.  Does this test makes any sense for hash
>> >> indexes considering LSN on hash index will always be zero (as hash indexes
>> >> are not WAL-logged)?  It seems to me that PageLSN check in
>> >> TestForOldSnapshot() will always return false which means that the error
>> >> "snapshot too old" won't be generated for hash indexes.
>> >>
>> >> Am I missing something here, if not, then I think we need a way to
>> >> prohibit pruning for hash indexes based on old_snapshot_threshold?
>> >
>> > What I mean to say here is prohibit pruning the relation which has hash
>> > index based on old_snapshot_threshold.
>>
>> Good spot; added to the open issues page.
>
> Uh, I have no idea how this would be fixed if the PageLSN is zero.  Do
> you?

Yes, I see three ways, the most obvious of which is what Amit
suggested -- don't do early vacuum on a table which has a hash index.

-- 
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, May 3, 2016 at 11:46 AM, Kevin Grittner <kgrittn@gmail.com> wrote:
>> Uh, I have no idea how this would be fixed if the PageLSN is zero.  Do
>> you?
>
> Yes, I see three ways, the most obvious of which is what Amit
> suggested -- don't do early vacuum on a table which has a hash index.

What do you mean by "early VACUUM"?  Amit suggested disabling
HOT-pruning, but HOT-pruning happens completely outside of VACUUM.  It
also happens inside VACUUM, so if we disabled HOT pruning, how could
we VACUUM at all?  Sorry, I am confused.

Doesn't this issue also affected indexes on any unlogged table?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, May 3, 2016 at 11:09 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Tue, May 3, 2016 at 11:46 AM, Kevin Grittner <kgrittn@gmail.com> wrote:
>>> Uh, I have no idea how this would be fixed if the PageLSN is zero.  Do
>>> you?
>>
>> Yes, I see three ways, the most obvious of which is what Amit
>> suggested -- don't do early vacuum on a table which has a hash index.
>
> What do you mean by "early VACUUM"?

Both vacuuming and hot-pruning adjust xmin based on calling
TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
Relation relation).  I'm talking about having that function, if all
other conditions for the override pass, checking for a hash index,
too.

> Amit suggested disabling
> HOT-pruning, but HOT-pruning happens completely outside of VACUUM.  It
> also happens inside VACUUM, so if we disabled HOT pruning, how could
> we VACUUM at all?  Sorry, I am confused.

I guess we were both talking a bit loosely since (as I mentioned
above) the function that adjusts the xmin is called for a vacuum or
pruning.  He mentioned one and I mentioned the other, but it's all
controlled by TransactionIdLimitedForOldSnapshots().

> Doesn't this issue also affected indexes on any unlogged table?

That's been covered all along.

--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, May 3, 2016 at 12:17 PM, Kevin Grittner <kgrittn@gmail.com> wrote:
> On Tue, May 3, 2016 at 11:09 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Tue, May 3, 2016 at 11:46 AM, Kevin Grittner <kgrittn@gmail.com> wrote:
>>>> Uh, I have no idea how this would be fixed if the PageLSN is zero.  Do
>>>> you?
>>>
>>> Yes, I see three ways, the most obvious of which is what Amit
>>> suggested -- don't do early vacuum on a table which has a hash index.
>>
>> What do you mean by "early VACUUM"?
>
> Both vacuuming and hot-pruning adjust xmin based on calling
> TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
> Relation relation).  I'm talking about having that function, if all
> other conditions for the override pass, checking for a hash index,
> too.
>
>> Amit suggested disabling
>> HOT-pruning, but HOT-pruning happens completely outside of VACUUM.  It
>> also happens inside VACUUM, so if we disabled HOT pruning, how could
>> we VACUUM at all?  Sorry, I am confused.
>
> I guess we were both talking a bit loosely since (as I mentioned
> above) the function that adjusts the xmin is called for a vacuum or
> pruning.  He mentioned one and I mentioned the other, but it's all
> controlled by TransactionIdLimitedForOldSnapshots().

OK, I see now: the basic idea here is that we can't prune based on the
newer XID unless the page LSN is guaranteed to advance whenever data
is removed.  Currently, we attempt to limit bloat in non-unlogged,
non-catalog tables.  You're saying we can instead attempt to limit
bloat only in non-unlogged, non-catalog tables without hash indexes,
and that will fix this issue.  Am I right?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, May 3, 2016 at 11:48 AM, Robert Haas <robertmhaas@gmail.com> wrote:

> OK, I see now: the basic idea here is that we can't prune based on the
> newer XID unless the page LSN is guaranteed to advance whenever data
> is removed.  Currently, we attempt to limit bloat in non-unlogged,
> non-catalog tables.  You're saying we can instead attempt to limit
> bloat only in non-unlogged, non-catalog tables without hash indexes,
> and that will fix this issue.  Am I right?

Right.

I was wondering whether there might be other avenues to the same
end, but that is the most obvious fix.  I'm hesitant to raise the
alternatives because people seem to have entered "panic mode", at
which point alternatives always look scary.

--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, May 3, 2016 at 11:48 AM, Robert Haas <robertmhaas@gmail.com> wrote:

> OK, I see now: the basic idea here is that we can't prune based on the
> newer XID unless the page LSN is guaranteed to advance whenever data
> is removed.  Currently, we attempt to limit bloat in non-unlogged,
> non-catalog tables.  You're saying we can instead attempt to limit
> bloat only in non-unlogged, non-catalog tables without hash indexes,
> and that will fix this issue.  Am I right?

As a first cut, something like the attached.

--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

<div dir="ltr">On Fri, May 6, 2016 at 12:45 AM, Amit Kapila <<a
href="mailto:amit.kapila16@gmail.com">amit.kapila16@gmail.com</a>>wrote:<br />> On Wed, May 4, 2016 at 7:48 PM,
KevinGrittner <<a href="mailto:kgrittn@gmail.com">kgrittn@gmail.com</a>> wrote:<br />>> On Tue, May 3, 2016
at11:48 AM, Robert Haas <<a href="mailto:robertmhaas@gmail.com">robertmhaas@gmail.com</a>> wrote:<br
/>>><br/>>>> OK, I see now: the basic idea here is that we can't prune based on the<br />>>>
newerXID unless the page LSN is guaranteed to advance whenever data<br />>>> is removed.  Currently, we
attemptto limit bloat in non-unlogged,<br />>>> non-catalog tables.  You're saying we can instead attempt to
limit<br/>>>> bloat only in non-unlogged, non-catalog tables without hash indexes,<br />>>> and that
willfix this issue.  Am I right?<br />>><br />>> As a first cut, something like the attached.<br />><br
/>>Patch looks good to me.  I have done some testing with hash and<br />> btree indexes and it works as
expected.<br/><br />Pushed with the addition of a paragraph to the docs regarding this<br />and some other situations
wherepeople have been unclear about what<br />to expect.<br /><br />--<br />Kevin Grittner<br />EDB: <a
href="http://www.enterprisedb.com">http://www.enterprisedb.com</a><br/>The Enterprise PostgreSQL Company<br /><br
/></div>