Thread: recovery testing for beta

On Thu, May 29, 2014 at 09:39:56AM -0700, Jeff Janes wrote:
> What features in 9.4 need more beta testing for recovery?
> 
> I've applied my partial-write testing harness to several scenarios in 9.4.  So
> far its found a recovery bug for gin indexes, a recovery bug for btree, a
> vacuum bug for btree indexes (with foreign keys, but that is not relevant to
> the bug), and nothing of interest for gist index, although it only tested
> "where text_array @@ to_tsquery(?)" queries.  
> 
> It also implicitly tested the xlog parallel write slots thing, as that is
> common code to all recovery.
> 
> I also applied the foreign key test retroactively to 9.3, and it quickly
> re-found the multixact bugs up until commit 9a57858f1103b89a5674.  The test was
> designed only with the knowledge that the bugs involved foreign keys and the
> consumption of multixacts.   I had no deeper knowledge of the details of those
> bugs when designing the test, so I have a reasonable amount of confidence that
> this could have found them in real time had I bothered to try to test the
> feature during the previous beta cycle.

Wow, that is impressive!  We are looking for a ways to find bugs like
the ones the appeared in 9.3.X, and it seems you might have found a way.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + Everyone has their own god. +

Jeff Janes wrote:

> One thing is that I want to find a way to drive multixact in fast forward,
> so that the freezing cycle gets a good workout. Currently I can't consume
> enough of them to make them wrap around within the time frame of a test.

IIRC I lobotomized it up by removing the XLogInsert() call.  That allows
you to generate large amounts of multixacts quickly.  In my laptop this
was able to do four or five wraparound cycles in two-three hours or so,
using the "burn multixact" utility here:
http://www.postgresql.org/message-id/20131231035913.GU22570@eldon.alvh.no-ip.org

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 05/29/2014 09:39 AM, Jeff Janes wrote:
> 
> I've applied my partial-write testing harness to several scenarios in
> 9.4.  So far its found a recovery bug for gin indexes, a recovery bug
> for btree, a vacuum bug for btree indexes (with foreign keys, but that
> is not relevant to the bug), and nothing of interest for gist index,
> although it only tested "where text_array @@ to_tsquery(?)" queries.

This is awesome.  Thanks for doing it!

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Jeff Janes wrote:
>> One thing is that I want to find a way to drive multixact in fast forward,
>> so that the freezing cycle gets a good workout. Currently I can't consume
>> enough of them to make them wrap around within the time frame of a test.

> IIRC I lobotomized it up by removing the XLogInsert() call.  That allows
> you to generate large amounts of multixacts quickly.  In my laptop this
> was able to do four or five wraparound cycles in two-three hours or so,
> using the "burn multixact" utility here:
> http://www.postgresql.org/message-id/20131231035913.GU22570@eldon.alvh.no-ip.org

Another possibility is to use pg_resetxlog to manually advance the
multixact counter to a point near wraparound.  I think you have to
manually create appropriate slru segment files as well when doing that
(someday we should hack pg_resetxlog to do that for you).  Still, it
might beat waiting hours to burn multixacts one by one.
        regards, tom lane

On 05/29/2014 07:39 PM, Jeff Janes wrote:
> It also implicitly tested the xlog parallel write slots thing, as that is
> common code to all recovery.

During development, I hit a lot of bugs in that patch by setting 
wal_buffers to 32kb (the minimum). Causes more backends to wait for each 
other, exposing deadlocks.

- Heikki

On Thu, May 29, 2014 at 09:39:56AM -0700, Jeff Janes wrote:
> I've applied my partial-write testing harness to several scenarios in 9.4.
>  So far its found a recovery bug for gin indexes, a recovery bug for btree,
> a vacuum bug for btree indexes (with foreign keys, but that is not relevant
> to the bug), and nothing of interest for gist index, although it only
> tested "where text_array @@ to_tsquery(?)" queries.
> 
> It also implicitly tested the xlog parallel write slots thing, as that is
> common code to all recovery.
> 
> I also applied the foreign key test retroactively to 9.3, and it quickly
> re-found the multixact bugs up until commit 9a57858f1103b89a5674.  The test
> was designed only with the knowledge that the bugs involved foreign keys
> and the consumption of multixacts.   I had no deeper knowledge of the
> details of those bugs when designing the test, so I have a reasonable
> amount of confidence that this could have found them in real time had I
> bothered to try to test the feature during the previous beta cycle.

Impressive.  This testing is of great value to the project.

-- 
Noah Misch
EnterpriseDB                                 http://www.enterprisedb.com

Hi,

On 2014-06-02 09:03:25 -0700, Jeff Janes wrote:
> On Fri, May 30, 2014 at 8:09 PM, Amit Kapila <amit.kapila16@gmail.com>
> wrote:
> 
> > On Sat, May 31, 2014 at 3:51 AM, Jeff Janes <jeff.janes@gmail.com> wrote:
> > > On Thu, May 29, 2014 at 8:15 PM, Amit Kapila <amit.kapila16@gmail.com>
> > wrote:
> > >> On Thu, May 29, 2014 at 10:09 PM, Jeff Janes <jeff.janes@gmail.com>
> > wrote:
> > >> >
> > >> > What features in 9.4 need more beta testing for recovery?
> > >>
> > >> Another feature which have interaction with recovery is reduced WAL
> > >> for Update operation:
> > >>
> > http://www.postgresql.org/message-id/E1WNqjM-0003cz-EL@gemulon.postgresql.org
> > >>
> > >> Commit: a3115f
> > >
> > >
> > > It looks like this is something which is "always on", so it should be
> > getting a good test without me taking any special steps.  But is there some
> > way, for example something in xlogdump, to assess how often this is getting
> > activated?
> >
> > Currently there is no simple way to get this information, but with
> > attached simple patch you can get this information by pg_xlogdump.
> >
> > It will give you information in below way:
> >
> > rmgr: Heap        len (rec/tot):     47/    79, tx:        690, lsn:
> > 0/0375B4A0,
> >  prev 0/0375B470, bkp: 0000, desc: hot_update: rel 1663/12125/24580; tid
> > 0/1 xma
> > x 690 ; new tid 0/2 xmax 0; *compressed tuple:* *suffix encoded*
> >
> > rmgr: Heap        len (rec/tot):     53/    85, tx:        691, lsn:
> > 0/0375B520,
> >  prev 0/0375B4F0, bkp: 0000, desc: hot_update: rel 1663/12125/24580; tid
> > 0/2 xma
> > x 691 ; new tid 0/3 xmax 0; *compressed tuple: prefix encoded*
> >
> > rmgr: Heap        len (rec/tot):     56/    88, tx:        692, lsn:
> > 0/0375B5A8,
> >  prev 0/0375B578, bkp: 0000, desc: hot_update: rel 1663/12125/24580; tid
> > 0/3 xma
> > x 692 ; new tid 0/4 xmax 0; *uncompressed tuple*
> >
> > I think this is useful information and can be even included in core
> > code.

I'd like to include something, but I think those are a bit long...

> Non-HOT updates can also be compressed, if they happen to land in the same
> page as the old version, so I copied that code into the non-HOT update
> section as well.

Right.

> GNU make does not realize that pg_xlogdump depends
> on src/backend/access/rmgrdesc/heapdesc.c.  (I don't know how or why it has
> that dependency, but changes did not take effect with a simple "make
> install") Is that a known issue?  Is there someway to fix it?

Hm. I can't reproduce this here. A simple 'touch heapdesc.c' triggers a
rebuild of pg_xlogdump for me. Could you include the make output?

> Also, pg_xlogdump -p .... insists on being given a start position.  I would
> be nice if it could just find the first file in the given directory.  Any
> reason it can't do that, other than just that no one implemented it yet?

It actually should accept getting a file passed instead of -s/-e.
pg_xlogdump [OPTION]... [STARTSEG [ENDSEG]]
That doesn't work for you?

Greetings,

Andres Freund

-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services

Hi,

On 2014-06-02 10:15:19 -0700, Jeff Janes wrote:
> > > Also, pg_xlogdump -p .... insists on being given a start position.  I
> > would
> > > be nice if it could just find the first file in the given directory.  Any
> > > reason it can't do that, other than just that no one implemented it yet?
> >
> > It actually should accept getting a file passed instead of -s/-e.
> > pg_xlogdump [OPTION]... [STARTSEG [ENDSEG]]
> > That doesn't work for you?
> >
> 
> The STARTSEG does not seem to be optional, unless you specify both -p and
> -s.  I don't think there is a very good command-line synopsis for such
> conditional dependencies--I think you are following the tradition here, in
> that if something is optional under any normal circumstances then it is put
> in brackets.

Maybe I have misunderstood what you actually want. You said:

> I would be nice if it could just find the first file in the given
> directory.

You mean it should scan the pg_xlog directory and find the numerically
oldest segment and start to decode from there?

> If I give for STARTSEG a path to the data directory or the pg_xlog
> directory instead of to a specific xlog file then I get bizarre error
> messages like:
>
> pg_xlogdump: FATAL:  could not find file "000000DA00C2C91500000080": No
> such file or directory
> (There is no timeline DA, nor C2C915 segments).  So STARTSEG cannot be a
> directory, which makes sense though the error message could be better.

Hm. Yes, that's clearly suboptimal. Will look if I can make it return a
more sensible error message.

> If I specify the full path of the first real log file as STARTSEG, it
> works.  But it is annoying to have to figure out what the first valid file
> in the directory is, then hope it hasn't changed while I type the command.
>  It is less annoying on an idle system or a snapshot, but it even there I'd
> rather not provide information that can be safely inferred.

I don't think it actually can be safely inferred in a trivial
manner... I guess a mode that reads the control file and starts with
values - including the timelineid - from there would be helpful.

Greetings,

Andres Freund

-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services

Jeff Janes wrote:

> GNU make does not realize that pg_xlogdump depends
> on src/backend/access/rmgrdesc/heapdesc.c.  (I don't know how or why it has
> that dependency, but changes did not take effect with a simple "make
> install") Is that a known issue?  Is there someway to fix it?

Uh, you're right, it's not.  How odd.  I think this might be an obscure
pgxs bug.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Hi Jeff,

On 2014-05-29 09:39:56 -0700, Jeff Janes wrote:
> What features in 9.4 need more beta testing for recovery?

Another thing I'd like to add to the list is wal_level=logical. Not such
much the logical decoding side, but that we haven't screwed up normal
crash recovery/wal replay.

> I also applied the foreign key test retroactively to 9.3, and it quickly
> re-found the multixact bugs up until commit 9a57858f1103b89a5674.

But you haven't found 1a917ae8610d44985f (master) / c0bd128c81c2b23a
(REL9_3_STABLE)? That's somewhat interesting...

Thanks for all the testing,

Andres Freund

-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services