Thread: PD_ALL_VISIBLE flag error on 9.0 alpha 4

PD_ALL_VISIBLE flag error on 9.0 alpha 4

From
Josh Berkus
Date:
All,

What I did:

1. Set up 9.0a4 doing SR replication with a 2nd 9.0a4
2. Ran pgbench for a while.
3. Aborted pgbench with Ctl-C
4. Changed vacuum_defer_cleanup_age in postgresql.conf and reloaded
5. Ran pgbench again, and got:

Sidney-Stratton:pg90 josh$ pgbench -c 2 -T 300 bench
starting vacuum...WARNING:  PD_ALL_VISIBLE flag was incorrectly set in
relation "pgbench_branches" page 0
WARNING:  PD_ALL_VISIBLE flag was incorrectly set in relation
"pgbench_branches" page 1
WARNING:  PD_ALL_VISIBLE flag was incorrectly set in relation
"pgbench_tellers" page 0
WARNING:  PD_ALL_VISIBLE flag was incorrectly set in relation
"pgbench_tellers" page 1

... not one I'm familiar with.  Issues?

--Josh Berkus

Re: PD_ALL_VISIBLE flag error on 9.0 alpha 4

From
Simon Riggs
Date:
On Tue, 2010-03-09 at 22:02 -0800, Josh Berkus wrote:

> 1. Set up 9.0a4 doing SR replication with a 2nd 9.0a4
> 2. Ran pgbench for a while.
> 3. Aborted pgbench with Ctl-C
> 4. Changed vacuum_defer_cleanup_age in postgresql.conf and reloaded
> 5. Ran pgbench again, and got:
>
> Sidney-Stratton:pg90 josh$ pgbench -c 2 -T 300 bench
> starting vacuum...WARNING:  PD_ALL_VISIBLE flag was incorrectly set in
> relation "pgbench_branches" page 0
> WARNING:  PD_ALL_VISIBLE flag was incorrectly set in relation
> "pgbench_branches" page 1
> WARNING:  PD_ALL_VISIBLE flag was incorrectly set in relation
> "pgbench_tellers" page 0
> WARNING:  PD_ALL_VISIBLE flag was incorrectly set in relation
> "pgbench_tellers" page 1

Understandable.

Time to remove vacuum_defer_cleanup_age, I think.

--
 Simon Riggs           www.2ndQuadrant.com

Re: PD_ALL_VISIBLE flag error on 9.0 alpha 4

From
Heikki Linnakangas
Date:
Simon Riggs wrote:
> On Tue, 2010-03-09 at 22:02 -0800, Josh Berkus wrote:
>
>> 1. Set up 9.0a4 doing SR replication with a 2nd 9.0a4
>> 2. Ran pgbench for a while.
>> 3. Aborted pgbench with Ctl-C
>> 4. Changed vacuum_defer_cleanup_age in postgresql.conf and reloaded
>> 5. Ran pgbench again, and got:
>>
>> Sidney-Stratton:pg90 josh$ pgbench -c 2 -T 300 bench
>> starting vacuum...WARNING:  PD_ALL_VISIBLE flag was incorrectly set in
>> relation "pgbench_branches" page 0
>> WARNING:  PD_ALL_VISIBLE flag was incorrectly set in relation
>> "pgbench_branches" page 1
>> WARNING:  PD_ALL_VISIBLE flag was incorrectly set in relation
>> "pgbench_tellers" page 0
>> WARNING:  PD_ALL_VISIBLE flag was incorrectly set in relation
>> "pgbench_tellers" page 1
>
> Understandable.
>
> Time to remove vacuum_defer_cleanup_age, I think.

Umm, so what's the bug?

--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

Re: PD_ALL_VISIBLE flag error on 9.0 alpha 4

From
Simon Riggs
Date:
On Wed, 2010-03-10 at 23:08 +0200, Heikki Linnakangas wrote:

> >
> > Time to remove vacuum_defer_cleanup_age, I think.
>
> Umm, so what's the bug?

Whether you call it a bug or just an annoyance is debatable, but the
source of it is clear. Given the lack of effectiveness, I propose
removing it.

Would you agree or disagree with the suggested removal?

--
 Simon Riggs           www.2ndQuadrant.com

Re: PD_ALL_VISIBLE flag error on 9.0 alpha 4

From
Tom Lane
Date:
Simon Riggs <simon@2ndQuadrant.com> writes:
>>> Time to remove vacuum_defer_cleanup_age, I think.
>>
>> Umm, so what's the bug?

> Whether you call it a bug or just an annoyance is debatable, but the
> source of it is clear.

Maybe to you, but the rest of us would like to know.

> Given the lack of effectiveness, I propose
> removing it.

I read Josh's recent report at
http://archives.postgresql.org/message-id/4B973C3F.9070501@agliodbs.com
to say that it's quite effective.  I think you're being way too hasty to
decide that it can just be dropped.

            regards, tom lane

Re: PD_ALL_VISIBLE flag error on 9.0 alpha 4

From
Simon Riggs
Date:
On Wed, 2010-03-10 at 17:55 -0500, Tom Lane wrote:
> Simon Riggs <simon@2ndQuadrant.com> writes:
> >>> Time to remove vacuum_defer_cleanup_age, I think.
> >>
> >> Umm, so what's the bug?
>
> > Whether you call it a bug or just an annoyance is debatable, but the
> > source of it is clear.
>
> Maybe to you, but the rest of us would like to know.

If vacuum_defer_cleanup_age is set higher this causes the xmin to go
backwards, leading to the "PD_ALL_VISIBLE flag was incorrectly set"
warning.

Having this false xmin move backwards doesn't endanger the standby,
since the xids arrive and are checked normally. If they stop arriving
that is fine.

Having the false xmin going backwards is not a serious issue on primary
because the actual xmin does not go backwards. No observer loses
information as a result of this, it is only about whether cleanup
records are generated later than normal, or not.

> > Given the lack of effectiveness, I propose
> > removing it.
>
> I read Josh's recent report at
> http://archives.postgresql.org/message-id/4B973C3F.9070501@agliodbs.com
> to say that it's quite effective.  I think you're being way too hasty to
> decide that it can just be dropped.

OK, that's enough to not remove it. I was aware of more negative
thoughts and conscious of my own feelings about it being a kluge.

--
 Simon Riggs           www.2ndQuadrant.com

Re: PD_ALL_VISIBLE flag error on 9.0 alpha 4

From
Josh Berkus
Date:
On 3/10/10 3:26 PM, Simon Riggs wrote:
> OK, that's enough to not remove it. I was aware of more negative
> thoughts and conscious of my own feelings about it being a kluge.

Well, it *is* a kludge, but it may be the best one for people who want
to use HS/SR to support web applications.  So I think we should work on
making it less kludgy.

Ultimately we're going to need publish-XID-to-master, but that's not
realistic for 9.0.

--Josh Berkus

Re: PD_ALL_VISIBLE flag error on 9.0 alpha 4

From
Simon Riggs
Date:
On Wed, 2010-03-10 at 17:12 -0800, Josh Berkus wrote:
> On 3/10/10 3:26 PM, Simon Riggs wrote:
> > OK, that's enough to not remove it. I was aware of more negative
> > thoughts and conscious of my own feelings about it being a kluge.
>
> Well, it *is* a kludge, but it may be the best one for people who want
> to use HS/SR to support web applications.  So I think we should work on
> making it less kludgy.

That's better, I was worried you'd gone all complimentary on me.

> Ultimately we're going to need publish-XID-to-master, but that's not
> realistic for 9.0.

Still some open items that I'm working on, but not that.

--
 Simon Riggs           www.2ndQuadrant.com

Re: PD_ALL_VISIBLE flag error on 9.0 alpha 4

From
Josh Berkus
Date:
> That's better, I was worried you'd gone all complimentary on me.

<grin>Never fear that!

Was that setting originally part of your design for HS?  If so, why did
you back off from it?

--Josh

Re: PD_ALL_VISIBLE flag error on 9.0 alpha 4

From
Simon Riggs
Date:
On Sat, 2010-03-13 at 11:29 -0800, Josh Berkus wrote:
> > That's better, I was worried you'd gone all complimentary on me.
>
> <grin>Never fear that!
>
> Was that setting originally part of your design for HS?  If so, why did
> you back off from it?

We all agreed its a kluge, that's why.

It's also my 3rd choice of solution behind fine-grained lock conflicts
(1st) which would avoid many issues and master/standby in lock step
(2nd).

Having said that I'm much in favour of providing a range of options and
then letting users tell us what works for them.

--
 Simon Riggs           www.2ndQuadrant.com

Re: PD_ALL_VISIBLE flag error on 9.0 alpha 4

From
Josh Berkus
Date:
> It's also my 3rd choice of solution behind fine-grained lock conflicts
> (1st) which would avoid many issues and master/standby in lock step
> (2nd).

Yeah, I just can't imagine you hunting down all of the corner cases for
fine-grained lock conflicts in time for 9.0.  Given what I've been
looking at, it seems like a LOT of work.

--Josh Berkus