Re: TAP output format in pg_regress - Mailing list pgsql-hackers

Attached is a new version of the pg_regress TAP patch which - per reviewer
feedback - does away with dual output formats and just converts the existing
output to be TAP complaint.  The idea was to keep it fairly human readable
while still be TAP compliant enough that running it with prove (with a tiny
Perl wrapper) works.  This version also strips away the verbose output which
these days isn't terribly useful and mainly consume vertical space.  Another
feature of the patch is to switch error logging to using the common frontend
logging rather than pg_regress rolling its own.  The output from pg_log_error
is emitted verbatim by prove as it's on stderr.

I based the support on the original TAP specification and not the new v13 or
v14 since it seemed unclear how well those are supported in testrunners.  If
v14 is adopted by testrunners there are some better functionality for reporting
errors that we could use, but for how this version seems a safer option.

A normal "make check" with this patch applied now looks like this:


+++ regress check in src/test/regress +++
# running on port 61696 with PID 57910
ok 1 - test_setup                       326 ms
ok 2 - tablespace                       401 ms
# parallel group (20 tests):  varchar char oid pg_lsn txid int4 regproc money int2 uuid float4 text name boolean int8
enumfloat8 bit numeric rangetypes 
ok 3 - boolean                          129 ms
ok 4 - char                              73 ms
ok 5 - name                             117 ms
ok 6 - varchar                           68 ms
<...snip...>
ok 210 - memoize                        137 ms
ok 211 - stats                          851 ms
# parallel group (2 tests):  event_trigger oidjoins
ok 212 - event_trigger                  138 ms
not ok 213 - oidjoins                   190 ms
ok 214 - fast_default                   149 ms
1..214
# 1 of 214 tests failed.
# The differences that caused some tests to fail can be viewed in the
# file "/<path>/src/test/regress/regression.diffs".  A copy of the test summary that you see
# above is saved in the file "/<path>/src/test/regress/regression.out".


The ending comment isn't currently shown when executed via prove as it has to
go to STDERR for that to work, and I'm not sure that's something we want to do
in the general case.  I don't expect running the pg_regress tests via prove is
going to be the common case.  How does the meson TAP ingestion handle
stdout/stderr?

And for the sake of completeness, even though we all know this by heart, a
similar run from the current output format looks like:


+++ regress check in src/test/regress +++
============== creating temporary instance            ==============
============== initializing database system           ==============
============== starting postmaster                    ==============
running on port 61696 with PID 61955
============== creating database "regression"         ==============
CREATE DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
============== running regression test queries        ==============
test test_setup                   ... ok          469 ms
test tablespace                   ... ok          415 ms
parallel group (20 tests):  varchar char oid name int2 pg_lsn int4 txid text uuid regproc boolean money float4 int8
float8bit enum numeric rangetypes 
     boolean                      ... ok          105 ms
     char                         ... ok           64 ms
     name                         ... ok           70 ms
     varchar                      ... ok           55 ms
<...snip...>
     memoize                      ... ok          149 ms
     stats                        ... ok          873 ms
parallel group (2 tests):  event_trigger oidjoins
     event_trigger                ... FAILED      142 ms
     oidjoins                     ... FAILED      208 ms
test fast_default                 ... ok          172 ms
============== shutting down postmaster               ==============

========================
 2 of 214 tests failed.
========================

The differences that caused some tests to fail can be viewed in the
file "/<path>/src/test/regress/regression.diffs".  A copy of the test summary that you see
above is saved in the file "/<path>/src/test/regress/regression.out".


There is a slight reduction in information, for example around tests run
serially vs in a parallel group.  This could perhaps be handled by marking the
test name in some way like "tablespace (serial) or prefixing with a symbol or
somerthing.  I can take a stab at that in case we think that level of detail is
important to preserve.

--
Daniel Gustafsson        https://vmware.com/