Direction for test frameworks: Perl TAP vs. Python/pytest - Mailing list pgsql-hackers

Hi,


We now have several pieces of test-infrastructure work in flight that 
point in
different directions, and I'd like the community to settle the direction 
before
more effort is expended on directions we might not want to follow.

Several things exist today:

1. My PostgreSQL::Test::Session work, which gives the Perl TAP framework
    in-process libpq sessions instead of forking psql per query. Part of the
    motivation is that BackgroundPsql is fragile and has a habit of 
swallowing
    errors; a real session object sidesteps that. It also means far less
    forking, so things run a good deal more efficiently -- especially on
    Windows, where process creation is expensive. The Perl framework is 
still
    gaining features. It currently relies on FFI::Platypus, and it's not 
clear
    that's available everywhere we build Postgres -- if not, we'd need 
to fall
    back on an XS wrapper. ("Pq Session object plus TAP test improvements",
    CF #6834 [1]; origin thread "Using LibPq in TAP tests via FFI" [2].)

2. Jelte Fennema-Nio's proposed minimal Python/pytest framework
    (src/test/pytest), aimed mainly at new tests. It already carries an
    in-process ctypes libpq layer of its own -- and note that on the Python
    side this needs no FFI module or XS wrapper at all, since ctypes is 
in the
    standard library and can bind to libpq natively. ("RFC: adding 
pytest as a
    supported test framework", CF #6045 [3].)

3. Two full ports of the existing TAP suite (~300 tests plus the 
framework) to
    that pytest shape, both validated against the existing suite. Greg 
Burd's [4]
    is a fully additive port: it leaves the existing Perl suite in place 
and runs
    a Python twin of each test beside it. Queries go through psql by default
    (matching safe_psql), with an opt-in in-process ctypes libpq layer for
    protocol-level tests. Mine [5] builds on Jelte's framework rather than
    standing apart from it: I've modularised and extended its libpq 
layer into a
    full Session along the lines of Test::Session, expanded the server and
    fixture helpers to cover replication, logical decoding, backups, and 
log and
    connection/auth checks, and added harness support for the SSL, LDAP,
    Kerberos and OAuth suites along with a pg_regress runner. I did this 
partly
    to show that we have an adequate basis for porting even if we never 
declare
    a flag day, and partly because, for what it's worth, I think the
    Test::Session model is a much more solid foundation for a port than the
    current framework is.

So the work supports more than one strategy. The question is which we want:

(A) Wholesale port with a flag day -- Python becomes *the* framework,
migrate the existing tests, stop accepting and eventually remove the 
Perl one.
One framework to learn and maintain, but a large disruptive change with real
dependency questions (which Python, which libpq binding, buildfarm 
minimums).

(B) Minimal Python framework for new tests, Perl preserved alongside.
Less disruptive, but commits us to two frameworks for some time.

(C) Something in between -- adopt the Python framework and port over 
just the
most fragile tests (the ones leaning on BackgroundPsql, say), leaving 
the rest
in Perl for now. This gets the worst pain points onto a sounder footing
without committing to a full migration up front.

(D) Keep both, but put each on a sounder footing: land the Test::Session 
work
so the Perl side stops depending on BackgroundPsql, and adopt the Python
framework alongside it for new tests. This addresses the fragility 
within Perl
rather than by porting, and doesn't force a migration either way.

If we land on (B), (C) or (D), I think we have to answer these questions:

- Do we implement new infrastructure features (e.g. the Session work) twice,
   or let the two frameworks drift?
- When an existing Perl test needs substantial new coverage, do we extend it
   in Perl or port it? "Port on touch" front-loads cost onto bug fixers.

Having shown a full port is feasible, my inclination is that the maintenance
argument favours converging on one framework -- but I'm wary of the 
disruption,
so I could see (D) as a sane and fairly safe option.

So how would people like to proceed?

cheers

andrew

[1] https://commitfest.postgresql.org/patch/6834/
[2] 
https://www.postgresql.org/message-id/997f67ac-cc02-4076-9fda-db465e89c622%40dunslane.net
[3] https://commitfest.postgresql.org/patch/6045/
[4] https://github.com/gburd/postgres/pull/24
[5] https://github.com/adunstan/pgdev/pull/5


--
Andrew Dunstan
EDB: https://www.enterprisedb.com