Thread: psql & regress tests

psql & regress tests

From
Peter Eisentraut
Date:
Okay, here is my semiofficial take on the situation:

* Use the psql from the 6.5.* distro to do regression tests until further
notice.


* Although the output format in the current psql is not under intensive
development anymore, I am not sure if I can guarantee a "freeze" soon.
Once in a while I find some sort of flaw in really strange query results;
the aim of the output is to be visually pleasing, not to provide an exact
match so something. Having said that, I do not expect any major changes to
take place anymore though.


* The other problem is *what* is actually printed, as opposed to the
peculiarities of the table format. This is still somewhat confusing even
to me and I am still fixing things so they make sense at the end.

Example:
***OLD
QUERY: CREATE TABLE BOOLTBL1 (f1 bool);
QUERY: INSERT INTO BOOLTBL1 (f1) VALUES ('t'::bool);
QUERY: INSERT INTO BOOLTBL1 (f1) VALUES ('True'::bool);
QUERY: INSERT INTO BOOLTBL1 (f1) VALUES ('true'::bool);
QUERY: SELECT '' AS t_3, BOOLTBL1.*;
t_3|f1
---+--  |t  |t  |t
(3 rows)

***CURRENT
CREATE TABLE BOOLTBL1 (f1 bool);
INSERT INTO BOOLTBL1 (f1) VALUES ('t'::bool);
INSERT INTO BOOLTBL1 (f1) VALUES ('True'::bool);
INSERT INTO BOOLTBL1 (f1) VALUES ('true'::bool);
-- BOOLTBL1 should be full of true's at this point
SELECT '' AS t_3, BOOLTBL1.*;t_3 | f1
-----+----    | t    | t    | t
(3 rows)

(In fact, it's so current, it's not even in CVS yet, thanks to some
problems pointed out by Jan.)

Yes, there actually is a reasoning behind all of this, I'm just not sure
right now what it was ;). If someone is interested, I can bore you with
the details.


* Since no one has picked up on my idea to run the tests directly on the
backend, I will keep reiterating this idea until someone shuts me up
:*) The idea would be to have a target "check" in the top level makefile
like this (conceptually):

check: allmkdir ./regressinitdb -l . -d ./regressfor i in test1 test2 test3 ...; do    postgres -D ./regress -E
template1\      < $(srcdir)/test/regress/sql/$i.sql \      >& output-$i.outdonefor i in test1 test2 test3 ...; do
cmpoutput-$i.out expected-$i.out    if [ $? == 1]; then        echo "Test $i failed."    else        echo "Test $i
passed."       rm -f output-$i.out    fidonerm -rf ./regress
 

Then you can do
./configure
make
make check
make install

Or am I missing something here? Of course this change would require some
work, but I'm just getting at the concept here.


Finally, I'd like to apologize for the extra trouble some must have had. I
can only offer to cooperate on anything that needs to be done.
-Peter

-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden





Re: [HACKERS] psql & regress tests

From
Bruce Momjian
Date:
> * Since no one has picked up on my idea to run the tests directly on the
> backend, I will keep reiterating this idea until someone shuts me up
> :*) The idea would be to have a target "check" in the top level makefile
> like this (conceptually):

Running the backend standalone does not use locking with other backends,
so it is dangerous.

--  Bruce Momjian                        |  http://www.op.net/~candle maillist@candle.pha.pa.us            |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: [HACKERS] psql & regress tests

From
"Ross J. Reedstrom"
Date:
On Thu, Nov 18, 1999 at 05:41:36PM -0500, Bruce Momjian wrote:
> > * Since no one has picked up on my idea to run the tests directly on the
> > backend, I will keep reiterating this idea until someone shuts me up
> > :*) The idea would be to have a target "check" in the top level makefile
> > like this (conceptually):
> 
> Running the backend standalone does not use locking with other backends,
> so it is dangerous.

Bruce, how dos this apply to Peter's suggestion?  We're talking about
_regression_ tests. Things to do after changing the code. Do you often
recompile, and run regression tests against a db with a (now out of date)
postmaster running against it? Do other developers?  I'd have thought a
complete shutdown/restart is part of the cycle anyway.  has to be if an
initdb is in there of course.  Checking to make sure a postmaster isn't
running could be added to Peter's script, just to be sure.

Ross
-- 
Ross J. Reedstrom, Ph.D., <reedstrm@rice.edu> 
NSBRI Research Scientist/Programmer
Computer and Information Technology Institute
Rice University, 6100 S. Main St.,  Houston, TX 77005


Re: [HACKERS] psql & regress tests

From
Bruce Momjian
Date:
> On Thu, Nov 18, 1999 at 05:41:36PM -0500, Bruce Momjian wrote:
> > > * Since no one has picked up on my idea to run the tests directly on the
> > > backend, I will keep reiterating this idea until someone shuts me up
> > > :*) The idea would be to have a target "check" in the top level makefile
> > > like this (conceptually):
> > 
> > Running the backend standalone does not use locking with other backends,
> > so it is dangerous.
> 
> Bruce, how dos this apply to Peter's suggestion?  We're talking about
> _regression_ tests. Things to do after changing the code. Do you often
> recompile, and run regression tests against a db with a (now out of date)
> postmaster running against it? Do other developers?  I'd have thought a
> complete shutdown/restart is part of the cycle anyway.  has to be if an
> initdb is in there of course.  Checking to make sure a postmaster isn't
> running could be added to Peter's script, just to be sure.

Regression tests are often run by end-users as part of the install.  I  
can imagine someone seeing a problem and running the regression tests
while having running backends to see if everything is still working.

--  Bruce Momjian                        |  http://www.op.net/~candle maillist@candle.pha.pa.us            |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: [HACKERS] psql & regress tests

From
wieck@debis.com (Jan Wieck)
Date:
>
> On Thu, Nov 18, 1999 at 05:41:36PM -0500, Bruce Momjian wrote:
> > > * Since no one has picked up on my idea to run the tests directly on the
> > > backend, I will keep reiterating this idea until someone shuts me up
> > > :*) The idea would be to have a target "check" in the top level makefile
> > > like this (conceptually):
> >
> > Running the backend standalone does not use locking with other backends,
> > so it is dangerous.
>
> Bruce, how dos this apply to Peter's suggestion?  We're talking about
> _regression_ tests. Things to do after changing the code. Do you often
> recompile, and run regression tests against a db with a (now out of date)
> postmaster running against it? Do other developers?  I'd have thought a
> complete shutdown/restart is part of the cycle anyway.  has to be if an
> initdb is in there of course.  Checking to make sure a postmaster isn't
> running could be added to Peter's script, just to be sure.

    I'm  actually  doing  some tests if the 'make check' would be
    possible.  I.e. doing a  complete  install  with  POSTGRESDIR
    below  the  regress  dir,  running initdb with the libdir and
    datadir pointing into there too, then starting new postmaster
    on a different port in background etc., etc., etc..

    That  would  make  it possible to do a complete check without
    doing a regular install at all.

    Will give some results soon.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #

Re: [HACKERS] psql & regress tests

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> * Since no one has picked up on my idea to run the tests directly on the
>> backend, I will keep reiterating this idea until someone shuts me up

> Running the backend standalone does not use locking with other backends,
> so it is dangerous.

It wouldn't be particularly "dangerous" if we assume that no one else is
accessing the regression database.  What it *would* be is less useful at
catching problems.  Standalone mode wouldn't test the cross-backend
interlocking code at all.

Admittedly, running a bunch of tests serially isn't a strong stress test
of cross-backend behavior, but it's not as content-free a check as you
might think.  On my machine, at least, the old backend is still around
doing shutdown for the first half-second or so while the next one is
running.

What I'd really like to see is some deliberate parallelism in some of
the regress tests...
        regards, tom lane


Re: [HACKERS] psql & regress tests

From
Bruce Momjian
Date:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> >> * Since no one has picked up on my idea to run the tests directly on the
> >> backend, I will keep reiterating this idea until someone shuts me up
> 
> > Running the backend standalone does not use locking with other backends,
> > so it is dangerous.
> 
> It wouldn't be particularly "dangerous" if we assume that no one else is
> accessing the regression database.  What it *would* be is less useful at
> catching problems.  Standalone mode wouldn't test the cross-backend
> interlocking code at all.

Any modifications to shared pg_ tables would be a problem.  Also, pg_log
and pg_variable locking is not happening in there either, is it?

--  Bruce Momjian                        |  http://www.op.net/~candle maillist@candle.pha.pa.us            |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: [HACKERS] psql & regress tests

From
wieck@debis.com (Jan Wieck)
Date:
Tom Lane wrote:

> Admittedly, running a bunch of tests serially isn't a strong stress test
> of cross-backend behavior, but it's not as content-free a check as you
> might think.  On my machine, at least, the old backend is still around
> doing shutdown for the first half-second or so while the next one is
> running.
>
> What I'd really like to see is some deliberate parallelism in some of
> the regress tests...

    It's  amusing how often we two have the same wishes or ideas.

    The run_check.sh script, I'm actually hacking on, would be  a
    replacement  for  the  regress.sh, started off from the 'make
    check'. And  from  the  first  try  I  added  syntax  to  the
    sql/tests  file  to  run  either  groups  of  tests  parallel
    intermixed with single tests sequentially.

    The new syntax will look like this:

        parallel    group1
            test    boolean
            test    char
            test    name
        endparallel

        test    varchar
        test    text
        test    strings

        parallel    group2
            test    int2
            test    int4
            test    int8
        endparallel

         .
         .
         .

    The above will run boolean, char and name parallel. After all
    three  terminated, it will check their output and continue to
    run varchar, text and strings sequentially, followed  by  the
    next parallel group.

    To  test  real  concurrency we might need to split up some or
    create  new  tests,  where  the  same  tables  are   accessed
    concurrently. But that wouldn't be hard to do I think.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #

Re: [HACKERS] psql & regress tests

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> It wouldn't be particularly "dangerous" if we assume that no one else is
>> accessing the regression database.  What it *would* be is less useful at
>> catching problems.  Standalone mode wouldn't test the cross-backend
>> interlocking code at all.

> Any modifications to shared pg_ tables would be a problem.  Also, pg_log
> and pg_variable locking is not happening in there either, is it?

Good point --- it wouldn't be just that database, but the whole
installation (data directory) that would have to be unused.  You really
wouldn't dare even have a postmaster running, at least not in the same
data directory.  But that could be made safe by using a nonstandard
location for the data directory for regression.

However, this is all beside the main point: we want the regress tests
to run in an environment as close as possible to the way Postgres is
normally used.  The more we hack up a special regress-test environment,
the less the tests reflect reality.

Aside from the cross-backend synchronization issue, I forgot to point
out a really obvious benefit: doing it the current way allows the regress
tests to help check the backend's frontend communication code, and
libpq, and psql itself.  We'd need some other way of testing all that
code if we switched to a standalone-backend regression test set.

What I *would* like to see is more support for running regress tests on
a not-yet-installed build, so people can test a fresh build before they
blow away their working installation.  This requires doing an initdb
into a temporary directory, starting a postmaster therein (using a
nonstandard port number), and running the tests there.  This is doable
by hand, of course, but it's tedious and error-prone even for an expert;
I think it's out of the question for a novice installer.  We ought to
offer a canned script to do it that way.
        regards, tom lane


Re: [HACKERS] psql & regress tests

From
wieck@debis.com (Jan Wieck)
Date:
Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
>
> > Any modifications to shared pg_ tables would be a problem.  Also, pg_log
> > and pg_variable locking is not happening in there either, is it?
>
> Good point --- it wouldn't be just that database, but the whole
> installation (data directory) that would have to be unused.  You really
> wouldn't dare even have a postmaster running, at least not in the same
> data directory.  But that could be made safe by using a nonstandard
> location for the data directory for regression.
>
> However, this is all beside the main point: we want the regress tests
> to run in an environment as close as possible to the way Postgres is
> normally used.  The more we hack up a special regress-test environment,
> the less the tests reflect reality.

    My new script actually does a

        make POSTGRESDIR=somewhere_else install
        PATH="somewhere_else/bin:$PATH"

    Then  it  initializes  a  database  below  there and starts a
    postmaster with the resulting data  directory,  listening  on
    port 65432.

    So  I  think  it's  very  close  to  a real live setup, while
    another "production" running installation isn't  affected  at
    all.

> Aside from the cross-backend synchronization issue, I forgot to point
> out a really obvious benefit: doing it the current way allows the regress
> tests to help check the backend's frontend communication code, and
> libpq, and psql itself.  We'd need some other way of testing all that
> code if we switched to a standalone-backend regression test set.
>
> What I *would* like to see is more support for running regress tests on
> a not-yet-installed build, so people can test a fresh build before they
> blow away their working installation.  This requires doing an initdb
> into a temporary directory, starting a postmaster therein (using a
> nonstandard port number), and running the tests there.  This is doable
> by hand, of course, but it's tedious and error-prone even for an expert;
> I think it's out of the question for a novice installer.  We ought to
> offer a canned script to do it that way.

    Right, right, right - I'm on it.

    The  ugly  detail,  I'm currently running into, is that there
    already seems to be a concurrency problem I  discovered  with
    my  testing.  Occationally I get this into the postmaster log
    for parallel executing tests:

ERROR:  Bad boolean external representation 'XXX'
FATAL 1:  SearchSysCache: recursive use of cache 10
FATAL 2:  elog: error in postmaster or backend startup, giving up!
pq_flush: send() failed: Broken pipe
Server process (pid 9791) exited with status 512 at Fri Nov 19 03:17:09 1999
Terminating any active server processes...

    It happens during the first parallel group of 11  tests.  Not
    allways, so it's timing critical. Outch.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #

Re: [HACKERS] psql & regress tests

From
Bruce Momjian
Date:
> ERROR:  Bad boolean external representation 'XXX'
> FATAL 1:  SearchSysCache: recursive use of cache 10
> FATAL 2:  elog: error in postmaster or backend startup, giving up!
> pq_flush: send() failed: Broken pipe
> Server process (pid 9791) exited with status 512 at Fri Nov 19 03:17:09 1999
> Terminating any active server processes...
> 
>     It happens during the first parallel group of 11  tests.  Not
>     allways, so it's timing critical. Outch.
> 

Now that we know numeric is working, can we make the test run faster in
the default mode?

--  Bruce Momjian                        |  http://www.op.net/~candle maillist@candle.pha.pa.us            |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: [HACKERS] psql & regress tests

From
Tom Lane
Date:
wieck@debis.com (Jan Wieck) writes:
>     I'm  actually  doing  some tests if the 'make check' would be
>     possible.  I.e. doing a  complete  install  with  POSTGRESDIR
>     below  the  regress  dir,  running initdb with the libdir and
>     datadir pointing into there too, then starting new postmaster
>     on a different port in background etc., etc., etc..
>     That  would  make  it possible to do a complete check without
>     doing a regular install at all.

Sounds great!  I was just griping elsewhere in this thread that we
needed that.

>     It's amusing how often we two have the same wishes or ideas.

Well, they're so obviously the Right Thing ;-)
        regards, tom lane


Re: [HACKERS] psql & regress tests

From
wieck@debis.com (Jan Wieck)
Date:
Bruce Momjian wrote:

> > ERROR:  Bad boolean external representation 'XXX'
> > FATAL 1:  SearchSysCache: recursive use of cache 10
> > FATAL 2:  elog: error in postmaster or backend startup, giving up!
> > pq_flush: send() failed: Broken pipe
> > Server process (pid 9791) exited with status 512 at Fri Nov 19 03:17:09 1999
> > Terminating any active server processes...
> >
> >     It happens during the first parallel group of 11  tests.  Not
> >     allways, so it's timing critical. Outch.

Hmmm,

    the  first  FATAL  is  emitted from catcache.c in line 988. I
    think that the cache->busy lives in shared memory  and  isn't
    protected  against  concurrent  usage, as it should be. Cache
    #10 is RELNAME. That really makes sense, because most of  the
    tests  I'm  running  parallel now issue CREATE TABLE commands
    first.


> Now that we know numeric is working, can we make the test run faster in
> the default mode?

    It is already down to 100 digits after the decimal point.   I
    don't  want  to  lower  it  too  much, but maybe 30 or 50 are
    enough too - no?


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #

Re: [HACKERS] psql & regress tests

From
Bruce Momjian
Date:
> > Now that we know numeric is working, can we make the test run faster in
> > the default mode?
> 
>     It is already down to 100 digits after the decimal point.   I
>     don't  want  to  lower  it  too  much, but maybe 30 or 50 are
>     enough too - no?

It is just taking longer than most other tests.  Seems 30 would be fine.
They can always run the bigtest.


--  Bruce Momjian                        |  http://www.op.net/~candle maillist@candle.pha.pa.us            |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: [HACKERS] psql & regress tests

From
Tom Lane
Date:
wieck@debis.com (Jan Wieck) writes:
>     The  ugly  detail,  I'm currently running into, is that there
>     already seems to be a concurrency problem I  discovered  with
>     my  testing.  Occationally I get this into the postmaster log
>     for parallel executing tests:

> ERROR:  Bad boolean external representation 'XXX'
> FATAL 1:  SearchSysCache: recursive use of cache 10
> FATAL 2:  elog: error in postmaster or backend startup, giving up!
> pq_flush: send() failed: Broken pipe
> Server process (pid 9791) exited with status 512 at Fri Nov 19 03:17:09 1999
> Terminating any active server processes...

>     It happens during the first parallel group of 11  tests.  Not
>     allways, so it's timing critical. Outch.

In other words, you've already exposed a bug!  Right on!

Commit the thing, so more eyes can look for the problem.  I expect
you have found a pre-existing backend bug, not a problem in your
new regress test scaffold.
        regards, tom lane


Re: [HACKERS] psql & regress tests

From
Tom Lane
Date:
wieck@debis.com (Jan Wieck) writes:
> Bruce Momjian wrote:
>> Now that we know numeric is working, can we make the test run faster in
>> the default mode?

>     It is already down to 100 digits after the decimal point.   I
>     don't  want  to  lower  it  too  much, but maybe 30 or 50 are
>     enough too - no?

Since multiply and so on are presumably O(N^2), cutting the precision
to 30 might cut the runtime by almost a factor of 10.

Jan probably has a better idea than the rest of us whether a test of
100, or 30, or 10 digits is likely to expose bugs that would not be
exposed by a test with less precision --- that depends on whether the
code has any internal behavioral dependence on the length of numeric
values.  The numeric test certainly is a lot slower than the others, so
I think it would be a good idea to trim the precision as much as we can.
Anyone who's actually touching the numeric code could and should run
the "bigtest", but the rest of us just want to know whether we've got
porting problems.  Seems like it shouldn't take 100-digit tests to
expose porting problems.
        regards, tom lane


Re: [HACKERS] psql & regress tests

From
wieck@debis.com (Jan Wieck)
Date:
Tom Lane wrote:

> wieck@debis.com (Jan Wieck) writes:
> > Bruce Momjian wrote:
> >> Now that we know numeric is working, can we make the test run faster in
> >> the default mode?
>
> >     It is already down to 100 digits after the decimal point.   I
> >     don't  want  to  lower  it  too  much, but maybe 30 or 50 are
> >     enough too - no?
>
> Since multiply and so on are presumably O(N^2), cutting the precision
> to 30 might cut the runtime by almost a factor of 10.
>
> Jan probably has a better idea than the rest of us whether a test of
> 100, or 30, or 10 digits is likely to expose bugs that would not be
> exposed by a test with less precision --- that depends on whether the

    I  created  a new default numeric test using numbers of range
    10,10 as input only. It doesn't save that much of time as you
    expect, but you're right anyways.

    Will  commit  it  later,  together with the new parallel test
    suite.

    BTW: The parallel problems  I  encountered  aren't  anything.
    Starting  the postmaster with -D... isn't the same as setting
    PGDATA environment - as it IMHO should be. It happened that I
    killed  the test-install postmaster, started with -D pointing
    into my temp dirs, with SIGKILL.  It corrupted the pg_control
    file of my default installation :-}

    And  if  you do not have an initialized data directory at the
    compiled in default location, postmaster doesn't startup with
    -D at all.

    Vadim, could you take a look at it?


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #

Re: [HACKERS] psql & regress tests

From
Peter Eisentraut
Date:
On 1999-11-19, Jan Wieck mentioned:

>     I'm  actually  doing  some tests if the 'make check' would be
>     possible.  I.e. doing a  complete  install  with  POSTGRESDIR
>     below  the  regress  dir,  running initdb with the libdir and
>     datadir pointing into there too, then starting new postmaster
>     on a different port in background etc., etc., etc..
> 
>     That  would  make  it possible to do a complete check without
>     doing a regular install at all.

Wasn't that exactly what my conceptual script intended to do? Great to see
some other people thinking in the same direction. The main point is to
have the tests run before any installation is done. Sounds like we're on
track . . .

-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden