test runner (was Re: SQL-standard function body) - Mailing list pgsql-hackers

Hi,

This started out as a reply to https://postgr.es/m/20210408170802.GA9392%40alvherre.pgsql
but it's independent enough to just start a new thread...

On 2021-04-08 13:08:02 -0400, Alvaro Herrera wrote:
> Yes, coverage.pg.org runs "make check-world".
>
> Maybe it would make sense to change that script, so that it runs the
> buildfarm's run_build.pl script instead of "make check-world".  That
> would make coverage.pg.org report what the buildfarm actually tests ...
> it would have made this problem a bit more obvious.

We desperately need to unify the different test run environments we
have. I did spent some time trying to do that, and ended up with it
being hard to do in a good way in the make / msvc environment. Not sure
that I took the right path, but I end up doing experimental port of the
buildsystem meson - which has a builtin test runner (working on all
platforms...).

    andres@awork3:/tmp$ ccache --clear
    andres@awork3:/tmp$ ~/src/meson/meson.py setup ~/src/postgresql /tmp/pg-meson --prefix=/tmp/pg-meson-install
    The Meson build system
    Version: 0.57.999
    Source dir: /home/andres/src/postgresql
    Build dir: /tmp/pg-meson
    Build type: native build
    Project name: postgresql
    Project version: 14devel
    ...
    Header <unistd.h> has symbol "fdatasync" : YES
    Header <fcntl.h> has symbol "F_FULLSYNC" : NO
    Checking for alignment of "short" : 2
    Checking for alignment of "int" : 4
    ...
    Configuring pg_config_ext.h using configuration
    Configuring pg_config.h using configuration
    Configuring pg_config_paths.h using configuration
    Program sed found: YES (/usr/bin/sed)
    Build targets in project: 116

    Found ninja-1.10.1 at /usr/bin/ninja
    ...

    andres@awork3:/tmp/pg-meson$ time ninja
    [10/1235] Generating snowball_create with a custom command
    Generating tsearch script...............................
    [41/1235] Generating generated_catalog_headers with a custom command
    [1235/1235] Linking target contrib/test_decoding/test_decoding.so

    real    0m10.752s
    user    3m47.020s
    sys    0m50.281s

    ...
    andres@awork3:/tmp/pg-meson$ time ninja
    [1/1] Generating test clean with a custom command

    real    0m0.085s
    user    0m0.068s
    sys    0m0.016s
    ...

    andres@awork3:/tmp/pg-meson$ time ~/src/meson/meson.py install --quiet
    ninja: Entering directory `.'

    real    0m0.541s
    user    0m0.412s
    sys    0m0.130s

    ...

    andres@awork3:/tmp/pg-meson$ ninja test
    [1/2] Running all tests.
     1/74 postgresql:setup / temp_install                                                         OK
0.52s
     2/74 postgresql:setup / cleanup_old                                                          OK
0.01s
     3/74 postgresql:tap+pg_archivecleanup / pg_archivecleanup/t/010_pg_archivecleanup.pl         OK
0.29s  42 subtests passed
 
     4/74 postgresql:tap+pg_checksums / pg_checksums/t/001_basic.pl                               OK
0.27s  8 subtests passed
 
     5/74 postgresql:tap+pg_config / pg_config/t/001_pg_config.pl                                 OK
0.26s  20 subtests passed
 
    ...
    68/74 postgresql:tap+pg_dump / pg_dump/t/002_pg_dump.pl                           OK               28.26s   6408
subtestspassed
 
    ...
    74/74 postgresql:isolation / pg_isolation_regress                                             OK
114.91s


    Ok:                 74
    Expected Fail:      0
    Fail:               0
    Unexpected Pass:    0
    Skipped:            0
    Timeout:            0

    Full log written to /tmp/pg-meson/meson-logs/testlog.txt


And in cases of failures it'll show the failure when it happens
(including the command to rerun just that test, without the harness in
between), and then a summary at the end:

    61/74 postgresql:tap+pg_verifybackup / pg_verifybackup/t/003_corruption.pl                    OK
10.65s  44 subtests passed
 
    49/74 postgresql:tap+recovery / recovery/t/019_replslot_limit.pl                              ERROR
7.53s  exit status 1
 
    >>> MALLOC_PERTURB_=16
PATH=/tmp/pg-meson/tmp_install///usr/local/bin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/home/andres/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin
PG_REGRESS=/tmp/pg-meson/src/test/regress/pg_regressREGRESS_SHLIB=/tmp/pg-meson/src/test/regress/regress.so
LD_LIBRARY_PATH=/tmp/pg-meson/tmp_install///usr/local/lib/x86_64-linux-gnu
/home/andres/src/postgresql/src/tools/testwrap/tmp/pg-meson recovery t/019_replslot_limit.pl perl -I
/home/andres/src/postgresql/src/test/perl-I /home/andres/src/postgresql/src/test/recovery
/home/andres/src/postgresql/src/test/recovery/t/019_replslot_limit.pl
    ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀
―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
    stderr:
    #   Failed test 'check that required WAL segments are still available'
    #   at /home/andres/src/postgresql/src/test/recovery/t/019_replslot_limit.pl line 168.
    # Looks like you failed 1 test of 14.

    (test program exited with status code 1)

―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

    62/74 postgresql:tap+pg_basebackup / pg_basebackup/t/010_dump_connstr.pl                      OK
11.59s  14 subtests passed
 
    ...
    74/74 postgresql:isolation / pg_isolation_regress                                             OK
112.26s

    Summary of Failures:

    49/74 postgresql:tap+recovery / recovery/t/019_replslot_limit.pl                      ERROR             7.53s
exitstatus 1
 


    Ok:                 73
    Expected Fail:      0
    Fail:               1
    Unexpected Pass:    0
    Skipped:            0
    Timeout:            0

    Full log written to /tmp/pg-meson/meson-logs/testlog.txt
    FAILED: meson-test
    /usr/bin/python3 -u /home/andres/src/meson/meson.py test --no-rebuild --print-errorlogs
    ninja: build stopped: subcommand failed.

It's hard to convey just how much nicer it is to see a progress report
during the test, see the failing tests at the end, without needing to
wade through reams of log output.  The output of the individual tests is
in testlog.txt referenced above.

One can get a list of tests and then also just run subsets of them:

    andres@awork3:/tmp/pg-meson$ ~/src/meson/meson.py test --list
    postgresql:setup / temp_install
    postgresql:setup / cleanup_old
    postgresql:isolation / pg_isolation_regress
    postgresql:regress / pg_regress
    postgresql:tap+initdb / initdb/t/001_initdb.pl

Can run "suites" of tests:
    ~/src/meson/meson.py test --suite setup --suite recovery

Can run individual tests:
    ~/src/meson/meson.py test recovery/t/008_fsm_truncation.pl


Obviously all very far from being ready, but this seemed like a good
enough excuse to mention it ;)

Greetings,

Andres Freund