Re: RFC: adding pytest as a supported test framework - Mailing list pgsql-hackers

On Wed, 12 Jun 2024 at 21:07, Robert Haas <robertmhaas@gmail.com> wrote:
> Yeah, I don't think depending on psycopg2 is practical at all. We can
> either shell out to psql like we do now, or we can use something like
> CFFI.

Quick clarification I meant psycopg3, not psycopg2. And I'd very much
like to avoid using psql for sending queries, luckily CFFI in python
is very good.

> Many of
> those improvements could be made on top of the Perl framework we have
> today, and some of them have been discussed, but nobody's done the
> work.

I agree it's not a technical issue. It is a people issue. There are
very few people skilled in Perl active in the community. And most of
those are very senior hackers that have much more important things to
do that make our Perl testing framework significantly better. And the
less senior people that might see improving tooling as a way to get
help out in the community, are try to stay away from Perl with a 10
foot pole. So the result is, nothing gets improved. Especially since
very few people outside our community improve this tooling either.

 I also don't understand the argument that assert a == b is some
> new and wonderful thing; I mean, you can already do is($a,$b,"test
> name") which *also* shows you the values when they don't match, and
> includes a test name, too!

Sure you can, if you know the function exists. And clearly not
everyone knows that it exists, as the quick grep below demonstrates:

❯ grep 'ok(.* == .*' **.pl | wc -l
41

But apart from the obvious syntax doing what you want, the output is
also much better when looking at a slightly more complex case. With
the following code:

def some_returning_func():
    return 1234

def some_func(val):
    if val > 100:
        return 100
    return val

def test_mytest():
    assert some_func(some_returning_func()) == 50

Pytest will show the following output

    def test_mytest():
>       assert some_func(some_returning_func()) == 50
E       assert 100 == 50
E        +  where 100 = some_func(1234)
E        +    where 1234 = some_returning_func()

I have no clue how you could get output that's even close to that
clear with Perl.

Another problem I run into is that, as you probably know, sometimes
you need to look at the postgres logs to find out what actually went
wrong. Currently the only way to find them (for me) is following the
following steps: hmm, let me figure out what that directory was called
again... ah okay it is build/testrun/pg_upgrade/001_basic/... okay
let's start opening log files that all have very similar names until
find the right one.

When a test in pytest fails it automatically outputs all stdout/stderr
that was outputted, and hides it on success. So for the PgBouncer test
suite. I simply send all the relevant log files to stdout, prefixed by
some capitalized identifying line with a few newlines around it.
Something like "PG_LOG: /path/to/actual/logfile". Then when a test
fails in my terminal window I can look at the files related to the
failed test instantly. This allows me to debug failures much faster.

A related thing that also doesn't help at all is that (afaik) seeing
any of the perl tap test output in your terminal requires running
`meson test` with the -v option, and then scrolling up past all the
super verbose output of successfully passing tests to find out what
exactly failed in the single test that failed. And if you don't want
to do that you have to navigate to the magic directory path (
build/testrun/pg_upgrade/001_basic/) of the specific tests to look at
the stdout file there... Which then turns out not to even be there if
you actually had a compilation failure in your perl script (which
happens very often to anyone that doesn't use perl often). So now you
have to scroll up anyway.

Pytest instead is very good at only showing output for the tests that
failed, and hiding pretty much all output for the tests that passed.