tests and meson - test names and file locations - Mailing list pgsql-hackers

Hi,

One of the things motivating me to work on the meson conversion is the ability
to run tests in an easily understandable way. Meson has a testrunner that
both allows to run all tests at once, and run subsets of tests.


= Test and testsuite naming =

Each test has a unique name, and 0-n labels (called 'suites'). One can run
tests by their name, or select tests by suites.

The way I've organized it so far is that tests are named like this:

main/pg_regress
main/isolation
recovery/t/018_wal_optimize.pl
pg_prewarm/t/001_basic.pl
test_shm_mq/pg_regress
psql/t/001_basic.pl

we could also name tests by their full path, but that makes the display very
unwieldy.


At the moment there's three suites differentiating by the type of test:
'pg_regress', 'isolation' and 'tap'. There's also a separate "axis" of suites,
describing what's being tested, e.g. 'main', 'test_decoding', 'recovery' etc.

That currently works out to each test having two suites, although I've
wondered about adding a 'contrib' suite as well.


Perhaps the pg_regress suite should just be named 'regress'? And perhaps the
t/ in the tap tests is superfluous - the display is already fairly wide?



= Log and Data locations =

To make things like the selection of log files for a specific test easier,
I've so far set it up so that test data and logs are stored in a separate
directory from the sources.

testrun/<main|recovery|...>/<testname>/<log|tmp_check|results...>

The runner now creates a test.start at the start of a test and either
test.success or test.failure at the end. That should make it pretty easy for
e.g. the buildfarm and CI to make the logs for a failed test easily
accessible. I've spent far too much time going through the ~hundred logs in
src/test/recovery/ that the buildfarm displays as one thing.


I really like having all the test data separately from the sources, but I get
that that's not what we've done so far. It's doable to just mirror the current
choice, but I don't think we should. But I won't push too hard against keeping
things the same.


I do wonder if we should put test data and log files in a separate directory
tree, but that'd be a bit more work probably.


Any comments on the above?


= Example outputs =

Here's an example output that you mostly should be able to make sense of now:

$ m test --print-errorlogs
ninja: Entering directory `/tmp/meson'
ninja: no work to do.
  1/242 postgresql:setup / tmp_install                                                              OK
0.33s
  2/242 postgresql:tap+pg_upgrade / pg_upgrade/t/001_basic.pl                                       OK
0.35s  8 subtests passed
 
  3/242 postgresql:tap+recovery / recovery/t/011_crash_recovery.pl                                  OK
2.67s  3 subtests passed
 
  4/242 postgresql:tap+recovery / recovery/t/013_crash_restart.pl                                   OK
2.91s  18 subtests passed
 
  5/242 postgresql:tap+recovery / recovery/t/014_unlogged_reinit.pl                                 OK
3.01s  23 subtests passed
 
  6/242 postgresql:tap+recovery / recovery/t/022_crash_temp_files.pl                                OK
3.16s  11 subtests passed
 
  7/242 postgresql:tap+recovery / recovery/t/016_min_consistency.pl                                 OK
3.43s  1 subtests passed
 
  8/242 postgresql:tap+recovery / recovery/t/021_row_visibility.pl                                  OK
3.46s  10 subtests passed
 
  9/242 postgresql:isolation+tcn / tcn/isolation                                                    OK
3.42s
 10/242 postgresql:tap+recovery / recovery/t/023_pitr_prepared_xact.pl                              OK
3.63s  1 subtests passed
 
...
241/242 postgresql:isolation+main / main/isolation                                                  OK
46.69s
242/242 postgresql:tap+pg_upgrade / pg_upgrade/t/002_pg_upgrade.pl                                  OK
57.00s  13 subtests passed
 

Ok:                 242
Expected Fail:      0
Fail:               0
Unexpected Pass:    0
Skipped:            0
Timeout:            0

Full log written to /tmp/meson/meson-logs/testlog.txt


The 'postgresql' is because meson supports subprojects (both to provide
dependencies if needed, and "real" subprojects), and their tests can be run at
once.


If a test fails it'll show the error output at the time of test:

39/242 postgresql:pg_regress+cube / cube/pg_regress                                                FAIL
3.74s  exit status 1
 
>>> REGRESS_SHLIB=/tmp/meson/src/test/regress/regress.so MALLOC_PERTURB_=44 PG_TEST_EXTRA='kerberos ldap ssl'
PG_REGRESS=/tmp/meson/src/test/regress/pg_regress
PATH=/tmp/meson/tmp_install/tmp/meson-install/bin:/tmp/meson/contrib/cube:/home/andres/bin/perl5/bin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/games
/usr/bin/python3/home/andres/src/postgresql/src/tools/testwrap --srcdir /home/andres/src/postgresql/contrib/cube
--basedir/tmp/meson --builddir /tmp/meson/contrib/cube --testgroup cube --testname pg_regress
/tmp/meson/src/test/regress/pg_regress--temp-instance /tmp/meson/testrun/cube/pg_regress/tmp_check --inputdir
/home/andres/src/postgresql/contrib/cube--expecteddir /home/andres/src/postgresql/contrib/cube --outputdir
/tmp/meson/testrun/cube/pg_regress--bindir '' --dlpath /tmp/meson/contrib/cube --max-concurrent-tests=20 --port=40012
cubecube_sci
 
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀
―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
# executing test in /tmp/meson/testrun/cube/pg_regress group cube test pg_regress, builddir /tmp/meson/contrib/cube
============== creating temporary instance            ==============
============== initializing database system           ==============
============== starting postmaster                    ==============
running on port 40012 with PID 354981
============== creating database "regression"         ==============
CREATE DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
============== running regression test queries        ==============
test cube                         ... FAILED      418 ms
test cube_sci                     ... ok           16 ms
============== shutting down postmaster               ==============

======================
 1 of 2 tests failed.
======================

The differences that caused some tests to fail can be viewed in the
file "/tmp/meson/testrun/cube/pg_regress/regression.diffs".  A copy of the test summary that you see
above is saved in the file "/tmp/meson/testrun/cube/pg_regress/regression.out".

# test failed

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

 40/242 postgresql:pg_regress+citext / citext/pg_regress                                            OK
3.77s
...

A list of failed tests is listed at the end of the test:


Summary of Failures:

 39/242 postgresql:pg_regress+cube / cube/pg_regress                                       FAIL              3.74s
exitstatus 1
 

Ok:                 241
Expected Fail:      0
Fail:               1
Unexpected Pass:    0
Skipped:            0
Timeout:            0

Full log written to /tmp/meson/meson-logs/testlog.txt


Greetings,

Andres Freund