Thread: psql & regress tests
Okay, here is my semiofficial take on the situation:
* Use the psql from the 6.5.* distro to do regression tests until further
notice.
* Although the output format in the current psql is not under intensive
development anymore, I am not sure if I can guarantee a "freeze" soon.
Once in a while I find some sort of flaw in really strange query results;
the aim of the output is to be visually pleasing, not to provide an exact
match so something. Having said that, I do not expect any major changes to
take place anymore though.
* The other problem is *what* is actually printed, as opposed to the
peculiarities of the table format. This is still somewhat confusing even
to me and I am still fixing things so they make sense at the end.
Example:
***OLD
QUERY: CREATE TABLE BOOLTBL1 (f1 bool);
QUERY: INSERT INTO BOOLTBL1 (f1) VALUES ('t'::bool);
QUERY: INSERT INTO BOOLTBL1 (f1) VALUES ('True'::bool);
QUERY: INSERT INTO BOOLTBL1 (f1) VALUES ('true'::bool);
QUERY: SELECT '' AS t_3, BOOLTBL1.*;
t_3|f1
---+-- |t |t |t
(3 rows)
***CURRENT
CREATE TABLE BOOLTBL1 (f1 bool);
INSERT INTO BOOLTBL1 (f1) VALUES ('t'::bool);
INSERT INTO BOOLTBL1 (f1) VALUES ('True'::bool);
INSERT INTO BOOLTBL1 (f1) VALUES ('true'::bool);
-- BOOLTBL1 should be full of true's at this point
SELECT '' AS t_3, BOOLTBL1.*;t_3 | f1
-----+---- | t | t | t
(3 rows)
(In fact, it's so current, it's not even in CVS yet, thanks to some
problems pointed out by Jan.)
Yes, there actually is a reasoning behind all of this, I'm just not sure
right now what it was ;). If someone is interested, I can bore you with
the details.
* Since no one has picked up on my idea to run the tests directly on the
backend, I will keep reiterating this idea until someone shuts me up
:*) The idea would be to have a target "check" in the top level makefile
like this (conceptually):
check: allmkdir ./regressinitdb -l . -d ./regressfor i in test1 test2 test3 ...; do postgres -D ./regress -E
template1\ < $(srcdir)/test/regress/sql/$i.sql \ >& output-$i.outdonefor i in test1 test2 test3 ...; do
cmpoutput-$i.out expected-$i.out if [ $? == 1]; then echo "Test $i failed." else echo "Test $i
passed." rm -f output-$i.out fidonerm -rf ./regress
Then you can do
./configure
make
make check
make install
Or am I missing something here? Of course this change would require some
work, but I'm just getting at the concept here.
Finally, I'd like to apologize for the extra trouble some must have had. I
can only offer to cooperate on anything that needs to be done.
-Peter
--
Peter Eisentraut Sernanders väg 10:115
peter_e@gmx.net 75262 Uppsala
http://yi.org/peter-e/ Sweden
> * Since no one has picked up on my idea to run the tests directly on the > backend, I will keep reiterating this idea until someone shuts me up > :*) The idea would be to have a target "check" in the top level makefile > like this (conceptually): Running the backend standalone does not use locking with other backends, so it is dangerous. -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
On Thu, Nov 18, 1999 at 05:41:36PM -0500, Bruce Momjian wrote: > > * Since no one has picked up on my idea to run the tests directly on the > > backend, I will keep reiterating this idea until someone shuts me up > > :*) The idea would be to have a target "check" in the top level makefile > > like this (conceptually): > > Running the backend standalone does not use locking with other backends, > so it is dangerous. Bruce, how dos this apply to Peter's suggestion? We're talking about _regression_ tests. Things to do after changing the code. Do you often recompile, and run regression tests against a db with a (now out of date) postmaster running against it? Do other developers? I'd have thought a complete shutdown/restart is part of the cycle anyway. has to be if an initdb is in there of course. Checking to make sure a postmaster isn't running could be added to Peter's script, just to be sure. Ross -- Ross J. Reedstrom, Ph.D., <reedstrm@rice.edu> NSBRI Research Scientist/Programmer Computer and Information Technology Institute Rice University, 6100 S. Main St., Houston, TX 77005
> On Thu, Nov 18, 1999 at 05:41:36PM -0500, Bruce Momjian wrote: > > > * Since no one has picked up on my idea to run the tests directly on the > > > backend, I will keep reiterating this idea until someone shuts me up > > > :*) The idea would be to have a target "check" in the top level makefile > > > like this (conceptually): > > > > Running the backend standalone does not use locking with other backends, > > so it is dangerous. > > Bruce, how dos this apply to Peter's suggestion? We're talking about > _regression_ tests. Things to do after changing the code. Do you often > recompile, and run regression tests against a db with a (now out of date) > postmaster running against it? Do other developers? I'd have thought a > complete shutdown/restart is part of the cycle anyway. has to be if an > initdb is in there of course. Checking to make sure a postmaster isn't > running could be added to Peter's script, just to be sure. Regression tests are often run by end-users as part of the install. I can imagine someone seeing a problem and running the regression tests while having running backends to see if everything is still working. -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
>
> On Thu, Nov 18, 1999 at 05:41:36PM -0500, Bruce Momjian wrote:
> > > * Since no one has picked up on my idea to run the tests directly on the
> > > backend, I will keep reiterating this idea until someone shuts me up
> > > :*) The idea would be to have a target "check" in the top level makefile
> > > like this (conceptually):
> >
> > Running the backend standalone does not use locking with other backends,
> > so it is dangerous.
>
> Bruce, how dos this apply to Peter's suggestion? We're talking about
> _regression_ tests. Things to do after changing the code. Do you often
> recompile, and run regression tests against a db with a (now out of date)
> postmaster running against it? Do other developers? I'd have thought a
> complete shutdown/restart is part of the cycle anyway. has to be if an
> initdb is in there of course. Checking to make sure a postmaster isn't
> running could be added to Peter's script, just to be sure.
I'm actually doing some tests if the 'make check' would be
possible. I.e. doing a complete install with POSTGRESDIR
below the regress dir, running initdb with the libdir and
datadir pointing into there too, then starting new postmaster
on a different port in background etc., etc., etc..
That would make it possible to do a complete check without
doing a regular install at all.
Will give some results soon.
Jan
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#========================================= wieck@debis.com (Jan Wieck) #
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> * Since no one has picked up on my idea to run the tests directly on the
>> backend, I will keep reiterating this idea until someone shuts me up
> Running the backend standalone does not use locking with other backends,
> so it is dangerous.
It wouldn't be particularly "dangerous" if we assume that no one else is
accessing the regression database. What it *would* be is less useful at
catching problems. Standalone mode wouldn't test the cross-backend
interlocking code at all.
Admittedly, running a bunch of tests serially isn't a strong stress test
of cross-backend behavior, but it's not as content-free a check as you
might think. On my machine, at least, the old backend is still around
doing shutdown for the first half-second or so while the next one is
running.
What I'd really like to see is some deliberate parallelism in some of
the regress tests...
regards, tom lane
> Bruce Momjian <pgman@candle.pha.pa.us> writes: > >> * Since no one has picked up on my idea to run the tests directly on the > >> backend, I will keep reiterating this idea until someone shuts me up > > > Running the backend standalone does not use locking with other backends, > > so it is dangerous. > > It wouldn't be particularly "dangerous" if we assume that no one else is > accessing the regression database. What it *would* be is less useful at > catching problems. Standalone mode wouldn't test the cross-backend > interlocking code at all. Any modifications to shared pg_ tables would be a problem. Also, pg_log and pg_variable locking is not happening in there either, is it? -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
Tom Lane wrote:
> Admittedly, running a bunch of tests serially isn't a strong stress test
> of cross-backend behavior, but it's not as content-free a check as you
> might think. On my machine, at least, the old backend is still around
> doing shutdown for the first half-second or so while the next one is
> running.
>
> What I'd really like to see is some deliberate parallelism in some of
> the regress tests...
It's amusing how often we two have the same wishes or ideas.
The run_check.sh script, I'm actually hacking on, would be a
replacement for the regress.sh, started off from the 'make
check'. And from the first try I added syntax to the
sql/tests file to run either groups of tests parallel
intermixed with single tests sequentially.
The new syntax will look like this:
parallel group1
test boolean
test char
test name
endparallel
test varchar
test text
test strings
parallel group2
test int2
test int4
test int8
endparallel
.
.
.
The above will run boolean, char and name parallel. After all
three terminated, it will check their output and continue to
run varchar, text and strings sequentially, followed by the
next parallel group.
To test real concurrency we might need to split up some or
create new tests, where the same tables are accessed
concurrently. But that wouldn't be hard to do I think.
Jan
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#========================================= wieck@debis.com (Jan Wieck) #
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> It wouldn't be particularly "dangerous" if we assume that no one else is
>> accessing the regression database. What it *would* be is less useful at
>> catching problems. Standalone mode wouldn't test the cross-backend
>> interlocking code at all.
> Any modifications to shared pg_ tables would be a problem. Also, pg_log
> and pg_variable locking is not happening in there either, is it?
Good point --- it wouldn't be just that database, but the whole
installation (data directory) that would have to be unused. You really
wouldn't dare even have a postmaster running, at least not in the same
data directory. But that could be made safe by using a nonstandard
location for the data directory for regression.
However, this is all beside the main point: we want the regress tests
to run in an environment as close as possible to the way Postgres is
normally used. The more we hack up a special regress-test environment,
the less the tests reflect reality.
Aside from the cross-backend synchronization issue, I forgot to point
out a really obvious benefit: doing it the current way allows the regress
tests to help check the backend's frontend communication code, and
libpq, and psql itself. We'd need some other way of testing all that
code if we switched to a standalone-backend regression test set.
What I *would* like to see is more support for running regress tests on
a not-yet-installed build, so people can test a fresh build before they
blow away their working installation. This requires doing an initdb
into a temporary directory, starting a postmaster therein (using a
nonstandard port number), and running the tests there. This is doable
by hand, of course, but it's tedious and error-prone even for an expert;
I think it's out of the question for a novice installer. We ought to
offer a canned script to do it that way.
regards, tom lane
Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
>
> > Any modifications to shared pg_ tables would be a problem. Also, pg_log
> > and pg_variable locking is not happening in there either, is it?
>
> Good point --- it wouldn't be just that database, but the whole
> installation (data directory) that would have to be unused. You really
> wouldn't dare even have a postmaster running, at least not in the same
> data directory. But that could be made safe by using a nonstandard
> location for the data directory for regression.
>
> However, this is all beside the main point: we want the regress tests
> to run in an environment as close as possible to the way Postgres is
> normally used. The more we hack up a special regress-test environment,
> the less the tests reflect reality.
My new script actually does a
make POSTGRESDIR=somewhere_else install
PATH="somewhere_else/bin:$PATH"
Then it initializes a database below there and starts a
postmaster with the resulting data directory, listening on
port 65432.
So I think it's very close to a real live setup, while
another "production" running installation isn't affected at
all.
> Aside from the cross-backend synchronization issue, I forgot to point
> out a really obvious benefit: doing it the current way allows the regress
> tests to help check the backend's frontend communication code, and
> libpq, and psql itself. We'd need some other way of testing all that
> code if we switched to a standalone-backend regression test set.
>
> What I *would* like to see is more support for running regress tests on
> a not-yet-installed build, so people can test a fresh build before they
> blow away their working installation. This requires doing an initdb
> into a temporary directory, starting a postmaster therein (using a
> nonstandard port number), and running the tests there. This is doable
> by hand, of course, but it's tedious and error-prone even for an expert;
> I think it's out of the question for a novice installer. We ought to
> offer a canned script to do it that way.
Right, right, right - I'm on it.
The ugly detail, I'm currently running into, is that there
already seems to be a concurrency problem I discovered with
my testing. Occationally I get this into the postmaster log
for parallel executing tests:
ERROR: Bad boolean external representation 'XXX'
FATAL 1: SearchSysCache: recursive use of cache 10
FATAL 2: elog: error in postmaster or backend startup, giving up!
pq_flush: send() failed: Broken pipe
Server process (pid 9791) exited with status 512 at Fri Nov 19 03:17:09 1999
Terminating any active server processes...
It happens during the first parallel group of 11 tests. Not
allways, so it's timing critical. Outch.
Jan
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#========================================= wieck@debis.com (Jan Wieck) #
> ERROR: Bad boolean external representation 'XXX' > FATAL 1: SearchSysCache: recursive use of cache 10 > FATAL 2: elog: error in postmaster or backend startup, giving up! > pq_flush: send() failed: Broken pipe > Server process (pid 9791) exited with status 512 at Fri Nov 19 03:17:09 1999 > Terminating any active server processes... > > It happens during the first parallel group of 11 tests. Not > allways, so it's timing critical. Outch. > Now that we know numeric is working, can we make the test run faster in the default mode? -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
wieck@debis.com (Jan Wieck) writes:
> I'm actually doing some tests if the 'make check' would be
> possible. I.e. doing a complete install with POSTGRESDIR
> below the regress dir, running initdb with the libdir and
> datadir pointing into there too, then starting new postmaster
> on a different port in background etc., etc., etc..
> That would make it possible to do a complete check without
> doing a regular install at all.
Sounds great! I was just griping elsewhere in this thread that we
needed that.
> It's amusing how often we two have the same wishes or ideas.
Well, they're so obviously the Right Thing ;-)
regards, tom lane
Bruce Momjian wrote:
> > ERROR: Bad boolean external representation 'XXX'
> > FATAL 1: SearchSysCache: recursive use of cache 10
> > FATAL 2: elog: error in postmaster or backend startup, giving up!
> > pq_flush: send() failed: Broken pipe
> > Server process (pid 9791) exited with status 512 at Fri Nov 19 03:17:09 1999
> > Terminating any active server processes...
> >
> > It happens during the first parallel group of 11 tests. Not
> > allways, so it's timing critical. Outch.
Hmmm,
the first FATAL is emitted from catcache.c in line 988. I
think that the cache->busy lives in shared memory and isn't
protected against concurrent usage, as it should be. Cache
#10 is RELNAME. That really makes sense, because most of the
tests I'm running parallel now issue CREATE TABLE commands
first.
> Now that we know numeric is working, can we make the test run faster in
> the default mode?
It is already down to 100 digits after the decimal point. I
don't want to lower it too much, but maybe 30 or 50 are
enough too - no?
Jan
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#========================================= wieck@debis.com (Jan Wieck) #
> > Now that we know numeric is working, can we make the test run faster in > > the default mode? > > It is already down to 100 digits after the decimal point. I > don't want to lower it too much, but maybe 30 or 50 are > enough too - no? It is just taking longer than most other tests. Seems 30 would be fine. They can always run the bigtest. -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
wieck@debis.com (Jan Wieck) writes:
> The ugly detail, I'm currently running into, is that there
> already seems to be a concurrency problem I discovered with
> my testing. Occationally I get this into the postmaster log
> for parallel executing tests:
> ERROR: Bad boolean external representation 'XXX'
> FATAL 1: SearchSysCache: recursive use of cache 10
> FATAL 2: elog: error in postmaster or backend startup, giving up!
> pq_flush: send() failed: Broken pipe
> Server process (pid 9791) exited with status 512 at Fri Nov 19 03:17:09 1999
> Terminating any active server processes...
> It happens during the first parallel group of 11 tests. Not
> allways, so it's timing critical. Outch.
In other words, you've already exposed a bug! Right on!
Commit the thing, so more eyes can look for the problem. I expect
you have found a pre-existing backend bug, not a problem in your
new regress test scaffold.
regards, tom lane
wieck@debis.com (Jan Wieck) writes:
> Bruce Momjian wrote:
>> Now that we know numeric is working, can we make the test run faster in
>> the default mode?
> It is already down to 100 digits after the decimal point. I
> don't want to lower it too much, but maybe 30 or 50 are
> enough too - no?
Since multiply and so on are presumably O(N^2), cutting the precision
to 30 might cut the runtime by almost a factor of 10.
Jan probably has a better idea than the rest of us whether a test of
100, or 30, or 10 digits is likely to expose bugs that would not be
exposed by a test with less precision --- that depends on whether the
code has any internal behavioral dependence on the length of numeric
values. The numeric test certainly is a lot slower than the others, so
I think it would be a good idea to trim the precision as much as we can.
Anyone who's actually touching the numeric code could and should run
the "bigtest", but the rest of us just want to know whether we've got
porting problems. Seems like it shouldn't take 100-digit tests to
expose porting problems.
regards, tom lane
Tom Lane wrote:
> wieck@debis.com (Jan Wieck) writes:
> > Bruce Momjian wrote:
> >> Now that we know numeric is working, can we make the test run faster in
> >> the default mode?
>
> > It is already down to 100 digits after the decimal point. I
> > don't want to lower it too much, but maybe 30 or 50 are
> > enough too - no?
>
> Since multiply and so on are presumably O(N^2), cutting the precision
> to 30 might cut the runtime by almost a factor of 10.
>
> Jan probably has a better idea than the rest of us whether a test of
> 100, or 30, or 10 digits is likely to expose bugs that would not be
> exposed by a test with less precision --- that depends on whether the
I created a new default numeric test using numbers of range
10,10 as input only. It doesn't save that much of time as you
expect, but you're right anyways.
Will commit it later, together with the new parallel test
suite.
BTW: The parallel problems I encountered aren't anything.
Starting the postmaster with -D... isn't the same as setting
PGDATA environment - as it IMHO should be. It happened that I
killed the test-install postmaster, started with -D pointing
into my temp dirs, with SIGKILL. It corrupted the pg_control
file of my default installation :-}
And if you do not have an initialized data directory at the
compiled in default location, postmaster doesn't startup with
-D at all.
Vadim, could you take a look at it?
Jan
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#========================================= wieck@debis.com (Jan Wieck) #
On 1999-11-19, Jan Wieck mentioned: > I'm actually doing some tests if the 'make check' would be > possible. I.e. doing a complete install with POSTGRESDIR > below the regress dir, running initdb with the libdir and > datadir pointing into there too, then starting new postmaster > on a different port in background etc., etc., etc.. > > That would make it possible to do a complete check without > doing a regular install at all. Wasn't that exactly what my conceptual script intended to do? Great to see some other people thinking in the same direction. The main point is to have the tests run before any installation is done. Sounds like we're on track . . . -- Peter Eisentraut Sernanders väg 10:115 peter_e@gmx.net 75262 Uppsala http://yi.org/peter-e/ Sweden