Thread: RFC: adding pytest as a supported test framework
Hi all, For the v18 cycle, I would like to try to get pytest [1] in as a supported test driver, in addition to the current offerings. (I'm tempted to end the email there.) We had an unconference session at PGConf.dev [2] around this topic. There seemed to be a number of nodding heads and some growing momentum. So here's a thread to try to build wider consensus. If you have a competing or complementary test proposal in mind, heads up! == Problem Statement(s) == 1. We'd like to rerun a failing test by itself. 2. It'd be helpful to see _what_ failed without poring over logs. These two got the most nodding heads of the points I presented. (#1 received tongue-in-cheek applause.) I think most modern test frameworks are going to give you these things, but unfortunately we don't have them. Additionally, 3. Many would like to use modern developer tooling during testing (language servers! autocomplete! debuggers! type checking!) and we can't right now. 4. It'd be great to split apart client-side tests from server-side tests. Driving Postgres via psql all the time is fine for acceptance testing, but it becomes a big problem when you need to test how clients talk to servers with incompatible feature sets, or how a peer behaves when talking to something buggy. 5. Personally, I want to implement security features test-first (for high code coverage and other benefits), and our Perl-based tests are usually too cumbersome for that. == Why pytest? == From the small and biased sample at the unconference session, it looks like a number of people have independently settled on pytest in their own projects. In my opinion, pytest occupies a nice space where it solves some of the above problems for us, and it gives us plenty of tools to solve the other problems without too much pain. Problem 1 (rerun failing tests): One architectural roadblock to this in our Test::More suite is that tests depend on setup that's done by previous tests. pytest allows you to declare each test's setup requirements via pytest fixtures, letting the test runner build up the world exactly as it needs to be for a single isolated test. These fixtures may be given a "scope" so that multiple tests may share the same setup for performance or other reasons. Problem 2 (seeing what failed): pytest does this via assertion introspection and very detailed failure reporting. If you haven't seen this before, take a look at the pytest homepage [1]; there's an example of a full log. Problem 3 (modern tooling): We get this from Python's very active developer base. Problems 4 (splitting client and server tests) and 5 (making it easier to write tests first) aren't really Python- or pytest-specific, but I have done both quite happily in my OAuth work [3], and I've since adapted that suite multiple times to develop and test other proposals on this list, like LDAP/SCRAM, client encryption, direct SSL, and compression. Python's standard library has lots of power by itself, with very good documentation. And virtualenvs and better package tooling have made it much easier, IMO, to avoid the XKCD dependency tangle [4] of the 2010s. When it comes to third-party packages, which I think we're probably going to want in moderation, we would still need to discuss supply chain safety. Python is not as mature here as, say, Go. == A Plan == Even if everyone were on board immediately, there's a lot of work to do. I'd like to add pytest in a more probationary status, so we can iron out the inevitable wrinkles. My proposal would be: 1. Commit bare-bones support in our Meson setup for running pytest, so everyone can kick the tires independently. 2. Add a test for something that we can't currently exercise. 3. Port a test from a place where the maintenance is terrible, to see if we can improve it. If we hate it by that point, no harm done; tear it back out. Otherwise we keep rolling forward. Thoughts? Suggestions? Thanks, --Jacob [1] https://docs.pytest.org/ [2] https://wiki.postgresql.org/wiki/PGConf.dev_2024_Developer_Unconference#New_testing_frameworks [3] https://github.com/jchampio/pg-pytest-suite [4] https://xkcd.com/1987/
Hi! On Mon, Jun 10, 2024 at 9:46 PM Jacob Champion <jacob.champion@enterprisedb.com> wrote: > Thoughts? Suggestions? Thank you for working on this. Do you think you could re-use something from testgres[1] package? Links. 1. https://github.com/postgrespro/testgres ------ Regards, Alexander Korotkov Supabase
Hi, Just for context for the rest the email: I think we desperately need to move off perl for tests. The infrastructure around our testing is basically unmaintained and just about nobody that started doing dev stuff in the last 10 years learned perl. On 2024-06-10 11:46:00 -0700, Jacob Champion wrote: > 4. It'd be great to split apart client-side tests from server-side > tests. Driving Postgres via psql all the time is fine for acceptance > testing, but it becomes a big problem when you need to test how > clients talk to servers with incompatible feature sets, or how a peer > behaves when talking to something buggy. That seems orthogonal to using pytest vs something else? > == Why pytest? == > > From the small and biased sample at the unconference session, it looks > like a number of people have independently settled on pytest in their > own projects. In my opinion, pytest occupies a nice space where it > solves some of the above problems for us, and it gives us plenty of > tools to solve the other problems without too much pain. We might be able to alleviate that by simply abstracting it away, but I found pytest's testrunner pretty painful. Oodles of options that are not very well documented and that often don't work because they are very specific to some situations, without that being explained. > Problem 1 (rerun failing tests): One architectural roadblock to this > in our Test::More suite is that tests depend on setup that's done by > previous tests. pytest allows you to declare each test's setup > requirements via pytest fixtures, letting the test runner build up the > world exactly as it needs to be for a single isolated test. These > fixtures may be given a "scope" so that multiple tests may share the > same setup for performance or other reasons. OTOH, that's quite likely to increase overall test times very significantly. Yes, sometimes that can be avoided with careful use of various features, but often that's hard, and IME is rarely done rigiorously. > Problem 2 (seeing what failed): pytest does this via assertion > introspection and very detailed failure reporting. If you haven't seen > this before, take a look at the pytest homepage [1]; there's an > example of a full log. That's not really different than what the perl tap test stuff allows. We indeed are bad at utilizing it, but I'm not sure that switching languages will change that. I think part of the problem is that the information about what precisely failed is often much harder to collect when testing multiple servers interacting than when doing localized unit tests. I think we ought to invest a bunch in improving that, I'd hope that a lot of that work would be largely independent of the language the tests are written in. > Python's standard library has lots of power by itself, with very good > documentation. And virtualenvs and better package tooling have made it > much easier, IMO, to avoid the XKCD dependency tangle [4] of the > 2010s. Ugh, I think this is actually python's weakest area. There's about a dozen package managers and "python distributions", that are at best half compatible, and the documentation situation around this is *awful*. > When it comes to third-party packages, which I think we're > probably going to want in moderation, we would still need to discuss > supply chain safety. Python is not as mature here as, say, Go. What external dependencies are you imagining? > == A Plan == > > Even if everyone were on board immediately, there's a lot of work to > do. I'd like to add pytest in a more probationary status, so we can > iron out the inevitable wrinkles. My proposal would be: > > 1. Commit bare-bones support in our Meson setup for running pytest, so > everyone can kick the tires independently. > 2. Add a test for something that we can't currently exercise. > 3. Port a test from a place where the maintenance is terrible, to see > if we can improve it. > > If we hate it by that point, no harm done; tear it back out. Otherwise > we keep rolling forward. I think somewhere between 1 and 4 a *substantial* amount of work would be required to provide a bunch of the infrastructure that Cluster.pm etc provide. Otherwise we'll end up with a lot of copy pasted code between tests. Greetings, Andres Freund
On 2024-06-10 Mo 16:04, Andres Freund wrote:
Hi, Just for context for the rest the email: I think we desperately need to move off perl for tests. The infrastructure around our testing is basically unmaintained and just about nobody that started doing dev stuff in the last 10 years learned perl.
Andres,
I get that you don't like perl. But it's hard for me to take this terribly seriously. "desperately" seems like massive overstatement at best. As for what up and coming developers learn, they mostly don't learn C either, and that's far more critical to what we do.
I'm not sure what part of the testing infrastructure you think is unmaintained. For example, the last release of Test::Simple was all the way back on April 25.
Maybe there are some technical superiorities about what Jacob is proposing, enough for us to add it to our armory. I'll keep an open mind on that.
But let's not throw the baby out with the bathwater. Quite apart from anything else, a wholesale rework of the test infrastructure would make backpatching more painful.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On Mon, 10 Jun 2024 at 20:46, Jacob Champion <jacob.champion@enterprisedb.com> wrote: > For the v18 cycle, I would like to try to get pytest [1] in as a > supported test driver, in addition to the current offerings. Huge +1 from me (but I'm definitely biased here) > Thoughts? Suggestions? I think the most important thing is that we make it easy for people to use this thing, and use it in a "correct" way. I have met very few people that actually like writing tests, so I think it's very important to make the barrier to do so as low as possible. For the PgBouncer repo I created my own pytest based test suite more ~1.5 years ago now. I tried to make it as easy as possible to write tests there, and it has worked out quite well imho. I don't think it makes sense to copy all things I did there verbatim, because some of it is quite specific to testing PgBouncer. But I do think there's quite a few things that could probably be copied (or at least inspire what you do). Some examples: 1. helpers to easily run shell commands, most importantly setting check=True by default[1] 2. helper to get a free tcp port[2] 3. helper to check if the log contains a specific string[3] 4. automatically show PG logs on test failure[4] 5. helpers to easily run sql commands (psycopg interface isn't very user friendly imho for the common case)[5] 6. startup/teardown cleanup logic[6] [1]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L83-L131 [2]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L210-L233 [3]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L1125-L1143 [4]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L1075-L1103 [5]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L326-L338 [6]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L546-L642 On Mon, 10 Jun 2024 at 22:04, Andres Freund <andres@anarazel.de> wrote: > > Problem 1 (rerun failing tests): One architectural roadblock to this > > in our Test::More suite is that tests depend on setup that's done by > > previous tests. pytest allows you to declare each test's setup > > requirements via pytest fixtures, letting the test runner build up the > > world exactly as it needs to be for a single isolated test. These > > fixtures may be given a "scope" so that multiple tests may share the > > same setup for performance or other reasons. > > OTOH, that's quite likely to increase overall test times very > significantly. Yes, sometimes that can be avoided with careful use of various > features, but often that's hard, and IME is rarely done rigiorously. You definitely want to cache things like initdb and "pg_ctl start". But that's fairly easy to do with some startup/teardown logic. For PgBouncer I create a dedicated schema for each test that needs to create objects and automatically drop that schema at the end of the test[6] (including any temporary objects outside of schemas like users/replication slots). You can even choose not to clean up certain large schemas if they are shared across multiple tests. [6]: https://github.com/pgbouncer/pgbouncer/blob/3f791020fb017c570fcd2db390600a353f1cba0c/test/utils.py#L546-L642 > > Problem 2 (seeing what failed): pytest does this via assertion > > introspection and very detailed failure reporting. If you haven't seen > > this before, take a look at the pytest homepage [1]; there's an > > example of a full log. > > That's not really different than what the perl tap test stuff allows. We > indeed are bad at utilizing it, but I'm not sure that switching languages will > change that. It's not about allowing, it's about doing the thing that you want by default. The following code assert a == b will show you the actual values of both a and b when the test fails, instead of saying something like "false is not true". Ofcourse you can provide a message here too, like with perl its ok function, but even when you don't the output is helpful. > I think part of the problem is that the information about what precisely > failed is often much harder to collect when testing multiple servers > interacting than when doing localized unit tests. > > I think we ought to invest a bunch in improving that, I'd hope that a lot of > that work would be largely independent of the language the tests are written > in. Well, as you already noted no-one that started doing dev stuff in the last 10 years knows Perl nor wants to learn it. So a large part of the community tries to touch the current perl test suite as little as possible. I personally haven't tried to improve anything about our perl testing framework, even though I'm normally very much into improving developer tooling. > > Python's standard library has lots of power by itself, with very good > > documentation. And virtualenvs and better package tooling have made it > > much easier, IMO, to avoid the XKCD dependency tangle [4] of the > > 2010s. > > Ugh, I think this is actually python's weakest area. There's about a dozen > package managers and "python distributions", that are at best half compatible, > and the documentation situation around this is *awful*. I definitely agree this is Python its weakest area. But since venv is part of the python standard library it's much better. I have the following short blurb in PgBouncer its test README[7] and it has worked for all contributors so far: # create a virtual environment (only needed once) python3 -m venv env # activate the environment. You will need to activate this environment in # your shell every time you want to run the tests. (so it's needed once per # shell). source env/bin/activate # Install the dependencies (only needed once, or whenever extra dependencies # get added to requirements.txt) pip install -r requirements.txt [7]: https://github.com/pgbouncer/pgbouncer/blob/master/test/README.md > I think somewhere between 1 and 4 a *substantial* amount of work would be > required to provide a bunch of the infrastructure that Cluster.pm etc > provide. Otherwise we'll end up with a lot of copy pasted code between tests. Totally agreed, that we should have a fairly decent base to work on top of. I think we should at least port a few tests to show that the base has at least the most basic functionality.
On Mon, 10 Jun 2024 at 22:47, Andrew Dunstan <andrew@dunslane.net> wrote: > As for what up and coming developers learn, they mostly don't learn C either, and that's far more critical to what we do. I think many up and coming devs have at least touched C somewhere (e.g. in university). And because it's more critical to the project and also to many other low level projects, they don't mind learning it so much if they don't know it yet. But I, for example, try to write as few Perl tests as possible, because getting good at Perl has pretty much no use to me outside of writing tests for postgres. (I do personally think that official Rust support in Postgres would probably be a good thing, but that is a whole other discussion that I'd like to save for some other day) > But let's not throw the baby out with the bathwater. Quite apart from anything else, a wholesale rework of the test infrastructurewould make backpatching more painful. Backporting test improvements to decrease backporting pain is something we don't look badly upon afaict (Citus its test suite breaks semi-regularly on minor PG version updates due to some slight output changes introduced by e.g. an updated version of the isolationtester).
Hi, On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote: > > On 2024-06-10 Mo 16:04, Andres Freund wrote: > > Hi, > > > > > > Just for context for the rest the email: I think we desperately need to move > > off perl for tests. The infrastructure around our testing is basically > > unmaintained and just about nobody that started doing dev stuff in the last 10 > > years learned perl. > Andres, > > I get that you don't like perl. I indeed don't particularly like perl - but that's really not the main issue. I've already learned [some of] it. What is the main issue is that I've also watched several newer folks try to write tests in it, and it was not pretty. > But it's hard for me to take this terribly seriously. "desperately" seems > like massive overstatement at best. Shrug. > As for what up and coming developers learn, they mostly don't learn C > either, and that's far more critical to what we do. C is a a lot more useful to to them than perl. And it's actually far more widely known these days than perl. C does teach you some reasonably low-level-ish understanding of hardware. There are gazillions of programs written in C that we'll have to maintain for decades. I don't think that's comparably true for perl. > I'm not sure what part of the testing infrastructure you think is > unmaintained. For example, the last release of Test::Simple was all the way > back on April 25. IPC::Run is quite buggy and basically just maintained by Noah these days. Greetings, Andres Freund
On 2024-06-10 Mo 21:49, Andres Freund wrote:
Hi, On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote:On 2024-06-10 Mo 16:04, Andres Freund wrote:Hi, Just for context for the rest the email: I think we desperately need to move off perl for tests. The infrastructure around our testing is basically unmaintained and just about nobody that started doing dev stuff in the last 10 years learned perl.Andres, I get that you don't like perl.I indeed don't particularly like perl - but that's really not the main issue. I've already learned [some of] it. What is the main issue is that I've also watched several newer folks try to write tests in it, and it was not pretty.
Hmm. I've done webinars in the past about how to write TAP tests for PostgreSQL, maybe I need to beef that up some.
I'm not sure what part of the testing infrastructure you think is unmaintained. For example, the last release of Test::Simple was all the way back on April 25.IPC::Run is quite buggy and basically just maintained by Noah these days.
Yes, that's true. I think the biggest pain point is possibly the recovery tests.
Some time ago I did some work on wrapping libpq using the perl FFI module. It worked pretty well, and would mean we could probably avoid many uses of IPC::Run, and would probably be substantially more efficient (no fork required). It wouldn't avoid all uses of IPC::Run, though.
But my point was mainly that while a new framework might have value, I don't think we need to run out and immediately rewrite several hundred TAP tests. Let's pick the major pain points and address those.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On Mon, Jun 10, 2024 at 1:04 PM Andres Freund <andres@anarazel.de> wrote: > Just for context for the rest the email: I think we desperately need to move > off perl for tests. The infrastructure around our testing is basically > unmaintained and just about nobody that started doing dev stuff in the last 10 > years learned perl. Okay. Personally, I'm going to try to stay out of discussions around subtracting Perl and focus on adding Python, for a bunch of different reasons: - Tests aren't cheap, but in my experience, the maintenance-cost math for tests is a lot different than the math for implementations. - I don't personally care for Perl, but having tests in any form is usually better than not having them. - Trying to convince people to get rid of X while adding Y is a good way to make sure Y never happens. > On 2024-06-10 11:46:00 -0700, Jacob Champion wrote: > > 4. It'd be great to split apart client-side tests from server-side > > tests. Driving Postgres via psql all the time is fine for acceptance > > testing, but it becomes a big problem when you need to test how > > clients talk to servers with incompatible feature sets, or how a peer > > behaves when talking to something buggy. > > That seems orthogonal to using pytest vs something else? Yes, I think that's fair. It's going to be hard not to talk about "things that pytest+Python don't give us directly but are much easier to build" in all of this (and I tried to call that out in the next section, maybe belatedly). I think I'm going to have to convince both a group of people who want to ask "why pytest in particular?" and a group of people who ask "why isn't what we have good enough?" > > == Why pytest? == > > > > From the small and biased sample at the unconference session, it looks > > like a number of people have independently settled on pytest in their > > own projects. In my opinion, pytest occupies a nice space where it > > solves some of the above problems for us, and it gives us plenty of > > tools to solve the other problems without too much pain. > > We might be able to alleviate that by simply abstracting it away, but I found > pytest's testrunner pretty painful. Oodles of options that are not very well > documented and that often don't work because they are very specific to some > situations, without that being explained. Hm. There are a bunch of them, but I've never needed to go through the oodles of options. Anything in particular that caused problems? > > Problem 1 (rerun failing tests): One architectural roadblock to this > > in our Test::More suite is that tests depend on setup that's done by > > previous tests. pytest allows you to declare each test's setup > > requirements via pytest fixtures, letting the test runner build up the > > world exactly as it needs to be for a single isolated test. These > > fixtures may be given a "scope" so that multiple tests may share the > > same setup for performance or other reasons. > > OTOH, that's quite likely to increase overall test times very > significantly. Yes, sometimes that can be avoided with careful use of various > features, but often that's hard, and IME is rarely done rigiorously. Well, scopes are pretty front and center when you start building pytest fixtures, and the complicated longer setups will hopefully converge correctly early on and be reused everywhere else. I imagine no one wants to build cluster setup from scratch. On a slight tangent, is this not a problem today? I mean... part of my personal long-term goal is in increasing test hygiene, which is going to take some shifts in practice. As long as review keeps the quality of the tests fairly high, I see the inevitable "our tests take too long" problem as a good one. That's true no matter what framework we use, unless the framework is so bad that no one uses it and the runtime is trivial. If we're worried that people will immediately start exploding the runtime and no one will notice during review, maybe we can have some infrastructure flag how much a patch increased it? > > Problem 2 (seeing what failed): pytest does this via assertion > > introspection and very detailed failure reporting. If you haven't seen > > this before, take a look at the pytest homepage [1]; there's an > > example of a full log. > > That's not really different than what the perl tap test stuff allows. We > indeed are bad at utilizing it, but I'm not sure that switching languages will > change that. Jelte already touched on this, but I wanted to hammer on the point: If no one, not even the developers who chose and like Perl, is using Test::More in a way that's maintainable, I would prefer to use a framework that does maintainable things by default so that you have to try really hard to screw it up. It is possible to screw up `assert actual == expected`, but it takes more work than doing it the right way. > I think part of the problem is that the information about what precisely > failed is often much harder to collect when testing multiple servers > interacting than when doing localized unit tests. > > I think we ought to invest a bunch in improving that, I'd hope that a lot of > that work would be largely independent of the language the tests are written > in. We do a lot more acceptance testing than internal testing, which came up as a major complaint from me and others during the unconference. One of the reasons people avoid writing internal tests in Perl is because it's very painful to find a rhythm with Test::More. From experience test-driving the OAuth work, I'm *very* happy with the development cycle that pytest gave me. Other languages _could_ do that, sure. It's a simple matter of programming... > Ugh, I think this is actually python's weakest area. There's about a dozen > package managers and "python distributions", that are at best half compatible, > and the documentation situation around this is *awful*. So... don't support the half-compatible stuff? I thought this conversation was still going on with Windows Perl (ActiveState? Strawberry?) but everyone just seems to pick what works for them and move on to better things to do. Modern CPython includes pip and venv. Done. If someone comes to us with some horrible Anaconda setup wanting to know why their duct tape doesn't work, can't we just tell them no? > > When it comes to third-party packages, which I think we're > > probably going to want in moderation, we would still need to discuss > > supply chain safety. Python is not as mature here as, say, Go. > > What external dependencies are you imagining? The OAuth pytest suite makes extensive use of - psycopg, to easily drive libpq; - construct, for on-the-wire packet representations and manipulation; and - pyca/cryptography, for easy generation of certificates and manual crypto testing. I'd imagine each would need considerable discussion, if there is interest in doing the same things that I do with them. > I think somewhere between 1 and 4 a *substantial* amount of work would be > required to provide a bunch of the infrastructure that Cluster.pm etc > provide. Otherwise we'll end up with a lot of copy pasted code between tests. Possibly, yes. I think it depends on what you want to test first, and there's a green-field aspect of hope/anxiety/ennui, too. Are you trying to port the acceptance-test framework that we already have, or are you trying to build a framework that can handle the things we can't currently test? Will it be easier to refactor duplication into shared fixtures when the language doesn't encourage an infinite number of ways to do things? Or will we have to keep on top of it to avoid pain? --Jacob
On Mon, Jun 10, 2024 at 12:26 PM Alexander Korotkov <aekorotkov@gmail.com> wrote: > Thank you for working on this. > Do you think you could re-use something from testgres[1] package? Possibly? I think we're all coming at this with our own bags of tricks and will need to carve off pieces to port, contribute, or reimplement. Does testgres have something in particular you'd like to see the Postgres tests support? Thanks, --Jacob
On Mon, Jun 10, 2024 at 06:49:11PM -0700, Andres Freund wrote: > On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote: > > On 2024-06-10 Mo 16:04, Andres Freund wrote: > > > Just for context for the rest the email: I think we desperately need to move > > > off perl for tests. The infrastructure around our testing is basically > > > unmaintained and just about nobody that started doing dev stuff in the last 10 > > > years learned perl. > > As for what up and coming developers learn, they mostly don't learn C > > either, and that's far more critical to what we do. > > C is a a lot more useful to to them than perl. And it's actually far more > widely known these days than perl. If we're going to test in a non-Perl language, I'd pick C over Python. There would be several other unlikely-community-choice languages I'd pick over Python (C#, Java, C++). We'd need a library like today's Perl PostgreSQL::Test to make C-language tests nice, but the same would apply to any new language. I also want the initial scope to be the new language coexisting with the existing Perl tests. If a bulk translation ever happens, it should happen long after the debut of the new framework. That said, I don't much trust a human-written bulk language translation to go through without some tests accidentally ceasing to test what they test in Perl today.
On 2024-06-11 Tu 19:48, Noah Misch wrote: > On Mon, Jun 10, 2024 at 06:49:11PM -0700, Andres Freund wrote: >> On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote: >>> On 2024-06-10 Mo 16:04, Andres Freund wrote: >>>> Just for context for the rest the email: I think we desperately need to move >>>> off perl for tests. The infrastructure around our testing is basically >>>> unmaintained and just about nobody that started doing dev stuff in the last 10 >>>> years learned perl. >>> As for what up and coming developers learn, they mostly don't learn C >>> either, and that's far more critical to what we do. >> C is a a lot more useful to to them than perl. And it's actually far more >> widely known these days than perl. > If we're going to test in a non-Perl language, I'd pick C over Python. There > would be several other unlikely-community-choice languages I'd pick over > Python (C#, Java, C++). We'd need a library like today's Perl > PostgreSQL::Test to make C-language tests nice, but the same would apply to > any new language. Indeed. We've invested quite a lot of effort on that infrastructure. I guess people can learn from what we've done so a second language might be easier to support. (Java would be my pick from your unlikely set, but I can see the attraction of Python.) > > I also want the initial scope to be the new language coexisting with the > existing Perl tests. If a bulk translation ever happens, it should happen > long after the debut of the new framework. That said, I don't much trust a > human-written bulk language translation to go through without some tests > accidentally ceasing to test what they test in Perl today. +1 cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
On Wed, 12 Jun 2024 at 01:48, Noah Misch <noah@leadboat.com> wrote: > If we're going to test in a non-Perl language, I'd pick C over Python. There > would be several other unlikely-community-choice languages I'd pick over > Python (C#, Java, C++). My main goals of this thread are: 1. Allowing people to quickly write tests 2. Have those tests do what the writer intended them to do 3. Have good error reporting by default Those goals indeed don't necesitate python. But I think those are really hard to achieve with any C based framework, and probably with C++ too. Also manual memory management in tests seems to add tons of complexity for basically no benefit. I think C#, Java, Go, Rust, Kotlin, and Swift would be acceptable choices for me (and possibly some more). They allow some type of introspection, they have a garbage collector, and their general tooling is quite good. But I think a dynamically typed scripting language is much more fitting for writing tests like this. I love static typing for production code, but imho it really doesn't have much benefit for tests. As scripting languages go, the ones that are still fairly heavily in use are Javascript, Python, Ruby, and PHP. I think all of those could probably work, but my personal order of preference would be Python, Ruby, Javascript, PHP. Finally, I'm definitely biased towards using Python myself. But I think there's good reasons for that: 1. In the data space, that Postgres in, Python is very heavily used for analysis 2. Everyone coming out of university these days knows it to some extent 3. Multiple people in the community have been writing Postgres related tests in python and have enjoyed doing so (me[1], Jacob[2], Alexander[3]) What language people want to write tests in is obviously very subjective. And obviously we cannot allow every language for writing tests. But I think if ~25% of the community prefers to write their tests in Python. Then that should be enough of a reason to allow them to do so. TO CLARIFY: This thread is not a proposal to replace Perl with Python. It's a proposal to allow people to also write tests in Python. > I also want the initial scope to be the new language coexisting with the > existing Perl tests. If a bulk translation ever happens, it should happen > long after the debut of the new framework. That said, I don't much trust a > human-written bulk language translation to go through without some tests > accidentally ceasing to test what they test in Perl today. I definitely don't think we should rewrite all the tests that we have in Perl today into some other language. But I do think that whatever language we choose, that language should make it as least as easy to write tests, as easy to read them and as easy to see that they are testing the intended thing, as is currently the case for Perl. Rewriting a few Perl tests into the new language, even if not merging the rewrite, is a good way of validating that imho. PS. For PgBouncer I actually hand-rewrote all the tests that we had in bash (which is the worst testing language ever) in Python and doing so actually found more bugs in PgBouncer code that our bash tests wouldn't catch. So it's not necessarily the case that you lose coverage by rewriting tests. [1]: https://github.com/pgbouncer/pgbouncer/tree/master/test [2]: https://github.com/jchampio/pg-pytest-suite [3]: https://github.com/postgrespro/testgres
On Tue, Jun 11, 2024 at 5:31 PM Jacob Champion <jacob.champion@enterprisedb.com> wrote: > On Mon, Jun 10, 2024 at 12:26 PM Alexander Korotkov > <aekorotkov@gmail.com> wrote: > > Thank you for working on this. > > Do you think you could re-use something from testgres[1] package? > > Possibly? I think we're all coming at this with our own bags of tricks > and will need to carve off pieces to port, contribute, or reimplement. > Does testgres have something in particular you'd like to see the > Postgres tests support? Generally, testgres was initially designed as Python analogue of what we have in src/test/perl/PostgreSQL/Test. In particular its testgres.PostgresNode is analogue of PostgreSQL::Test::Cluster. It comes under PostgreSQL License. So, I wonder if we could revise it and fetch most part of it into our source tree. ------ Regards, Alexander Korotkov Supabase
On Wed, Jun 12, 2024 at 2:48 PM Alexander Korotkov <aekorotkov@gmail.com> wrote: > On Tue, Jun 11, 2024 at 5:31 PM Jacob Champion > <jacob.champion@enterprisedb.com> wrote: > > On Mon, Jun 10, 2024 at 12:26 PM Alexander Korotkov > > <aekorotkov@gmail.com> wrote: > > > Thank you for working on this. > > > Do you think you could re-use something from testgres[1] package? > > > > Possibly? I think we're all coming at this with our own bags of tricks > > and will need to carve off pieces to port, contribute, or reimplement. > > Does testgres have something in particular you'd like to see the > > Postgres tests support? > > Generally, testgres was initially designed as Python analogue of what > we have in src/test/perl/PostgreSQL/Test. In particular its > testgres.PostgresNode is analogue of PostgreSQL::Test::Cluster. It > comes under PostgreSQL License. So, I wonder if we could revise it > and fetch most part of it into our source tree. Plus testgres exists from 2016 and already have quite amount of use cases. This is what I quickly found on github. https://github.com/adjust/pg_querylog https://github.com/postgrespro/pg_pathman https://github.com/lanterndata/lantern https://github.com/orioledb/orioledb https://github.com/cbandy/pgtwixt https://github.com/OpenNTI/nti.testing https://github.com/postgrespro/pg_probackup https://github.com/postgrespro/rum ------ Regards, Alexander Korotkov Supabase
On Wed, 12 Jun 2024 at 01:48, Noah Misch <noah@leadboat.com> wrote: > If we're going to test in a non-Perl language, I'd pick C over Python. > <snip> > We'd need a library like today's Perl > PostgreSQL::Test to make C-language tests nice, but the same would apply to > any new language. P.P.S. We already write tests in C, we use it for testing libpq[1]. I'd personally definitely welcome a C library to make those tests nicer to write, because I've written a fair bit of those in the past and currently it's not very fun to do. [1]: https://github.com/postgres/postgres/blob/master/src/test/modules/libpq_pipeline/libpq_pipeline.c
> On Jun 12, 2024, at 6:40 AM, Jelte Fennema-Nio <postgres@jeltef.nl> wrote: > > I think C#, Java, Go, Rust, Kotlin, and Swift would be acceptable > choices for me (and possibly some more). They allow some type of > introspection, they have a garbage collector, and their general > tooling is quite good. > Having used Python for 15+ years and then abandoned it for all projects I would say the single most important points for a long term project like Postgres is, not necessarily in order, package stability, package depth, semantic versioning, available resources, and multiprocessor support. The reason I abandoned Python was for the constant API breaks in packages. Yes, python is a great language to teach in school for a one-time class project, but that is not Postgres’s use-case. Remember that Fortran and Pascal were the darlings for teaching in school prior to Python and no-one uses them any more. Yes Python innovates fast and is fashionable. But again, not Postgres’s use-case. I believe that anyone coming out of school these days would have a relatively easy transition to any of Go, Rust, Kotlin, Swift, etc. In other words, any of the modern languages. In addition, the language should scale well to multiprocessors, because parallel testing is becoming more important every day. If the Postgres project is going to pick a new language for testing, it should pick a language for the next 50 years based on the projects needs. Python is good for package depth and resource availability, but fails IMO in the other categories. My experience with python where the program flow can change because of non-visible characters is a terrible way to write robust long term maintainable code. Because of this most of the modern languages are going to be closer in style to Postgres’s C code base than Python.
Jelte Fennema-Nio: > As scripting languages go, the ones that are still fairly heavily in > use are Javascript, Python, Ruby, and PHP. I think all of those could > probably work, but my personal order of preference would be Python, > Ruby, Javascript, PHP. > > Finally, I'm definitely biased towards using Python myself. But I > think there's good reasons for that: > 1. In the data space, that Postgres in, Python is very heavily used for analysis > 2. Everyone coming out of university these days knows it to some extent > 3. Multiple people in the community have been writing Postgres related > tests in python and have enjoyed doing so (me[1], Jacob[2], > Alexander[3]) PostgREST also uses pytest for integration tests - and that was a very good decision compared to the bash based tests we had before. One more argument for Python compared to the other mentioned scripting languages: Python is already a development dependency via meson. None of the other 3 are. In a future where meson will be the only build system, we will have python "for free" already. Best, Wolfgang
Hi, On 2024-06-11 08:04:57 -0400, Andrew Dunstan wrote: > Some time ago I did some work on wrapping libpq using the perl FFI module. > It worked pretty well, and would mean we could probably avoid many uses of > IPC::Run, and would probably be substantially more efficient (no fork > required). It wouldn't avoid all uses of IPC::Run, though. FWIW, I'd *love* to see work on this continue. The reduction in test runtime on windows is substantial and would shorten the hack->CI->fail->hack loop a good bit shorter. And save money. > But my point was mainly that while a new framework might have value, I don't > think we need to run out and immediately rewrite several hundred TAP tests. Oh, yea. That's not at all feasible to just do in one go. Greetings, Andres Freund
On Wed, 12 Jun 2024 at 15:49, FWS Neil <neil@fairwindsoft.com> wrote: > I believe that anyone coming out of school these days would have a relatively > easy transition to any of Go, Rust, Kotlin, Swift, etc. In other words, any of > the modern languages. Agreed, which is why I said they'd be acceptable to me. But I think one important advantage of Python is that it's clear that many people in the community are willing to write tests in it. At PGConf.dev there were a lot of people in the unconference session about this. Also many people already wrote a Postgres testing framework for python, and are using it (see list of projects that Alexander shared). I haven't seen that level of willingness to write tests for any of those other languages (yet). > In addition, the language should scale well to > multiprocessors, because parallel testing is becoming more important every day. > <snip> > Python is good for package depth and resource availability, but fails IMO in the > other categories. You can easily pin packages or call a different function based on the version of the package, so I'm not sure what the problem is with package stability. Also chances are we'll pull in very little external packages and rely mostly on the stdlib (which is quite stable). Regarding parallelised running of tests, I agree that's very important. And indeed normally parallelism in python can be a pain (although async/await makes I/O parallelism a lot better at least). But running pytest tests in parallel is extremely easy by using pytest-xdist[1], so I don't really think there's an issue for this specific Python usecase. > My experience with python where the program flow can change > because of non-visible characters is a terrible way to write robust long term > maintainable code. Because of this most of the modern languages are going to be > closer in style to Postgres’s C code base than Python. I'm assuming this is about spaces vs curly braces for blocks? Now that we have auto formatters for every modern programming language I indeed prefer curly braces myself too. But honestly that's pretty much a tabs vs spaces discussion. [1]: https://pypi.org/project/pytest-xdist/ On Wed, 12 Jun 2024 at 15:49, FWS Neil <neil@fairwindsoft.com> wrote: > > > > On Jun 12, 2024, at 6:40 AM, Jelte Fennema-Nio <postgres@jeltef.nl> wrote: > > > > I think C#, Java, Go, Rust, Kotlin, and Swift would be acceptable > > choices for me (and possibly some more). They allow some type of > > introspection, they have a garbage collector, and their general > > tooling is quite good. > > > > Having used Python for 15+ years and then abandoned it for all projects I would > say the single most important points for a long term project like Postgres is, > not necessarily in order, package stability, package depth, semantic versioning, > available resources, and multiprocessor support. > > The reason I abandoned Python was for the constant API breaks in packages. Yes, > python is a great language to teach in school for a one-time class project, but > that is not Postgres’s use-case. Remember that Fortran and Pascal were the > darlings for teaching in school prior to Python and no-one uses them any more. > > Yes Python innovates fast and is fashionable. But again, not Postgres’s use-case. > > I believe that anyone coming out of school these days would have a relatively > easy transition to any of Go, Rust, Kotlin, Swift, etc. In other words, any of > the modern languages. In addition, the language should scale well to > multiprocessors, because parallel testing is becoming more important every day. > > If the Postgres project is going to pick a new language for testing, it should > pick a language for the next 50 years based on the projects needs. > > Python is good for package depth and resource availability, but fails IMO in the > other categories. My experience with python where the program flow can change > because of non-visible characters is a terrible way to write robust long term > maintainable code. Because of this most of the modern languages are going to be > closer in style to Postgres’s C code base than Python.
Hi, On 2024-06-11 07:28:23 -0700, Jacob Champion wrote: > On Mon, Jun 10, 2024 at 1:04 PM Andres Freund <andres@anarazel.de> wrote: > > Just for context for the rest the email: I think we desperately need to move > > off perl for tests. The infrastructure around our testing is basically > > unmaintained and just about nobody that started doing dev stuff in the last 10 > > years learned perl. > > Okay. Personally, I'm going to try to stay out of discussions around > subtracting Perl and focus on adding Python, for a bunch of different > reasons: I think I might have formulated my paragraph above badly - I didn't mean that we should move away from perl tests tomorrow, but that we need a path forward that allows folks to write tests without perl. > - Tests aren't cheap, but in my experience, the maintenance-cost math > for tests is a lot different than the math for implementations. At the moment they tend to be *more* expensive often, due to spurious failures. That's mostly not perl's fault, don't get me wrong, but us not having better infrastructure for testing complicated behaviour and/or testing things on a more narrow basis. > > > Problem 1 (rerun failing tests): One architectural roadblock to this > > > in our Test::More suite is that tests depend on setup that's done by > > > previous tests. pytest allows you to declare each test's setup > > > requirements via pytest fixtures, letting the test runner build up the > > > world exactly as it needs to be for a single isolated test. These > > > fixtures may be given a "scope" so that multiple tests may share the > > > same setup for performance or other reasons. > > > > OTOH, that's quite likely to increase overall test times very > > significantly. Yes, sometimes that can be avoided with careful use of various > > features, but often that's hard, and IME is rarely done rigiorously. > > Well, scopes are pretty front and center when you start building > pytest fixtures, and the complicated longer setups will hopefully > converge correctly early on and be reused everywhere else. I imagine > no one wants to build cluster setup from scratch. One (the?) prime source of state in our tap tests is the database. Realistically we can't just tear that one down and reset it between tests without causing the test times to explode. So we'll have to live with some persistent state. > On a slight tangent, is this not a problem today? It is, but that doesn't mean making it even bigger is unproblematic :) > > I think part of the problem is that the information about what precisely > > failed is often much harder to collect when testing multiple servers > > interacting than when doing localized unit tests. > > > > I think we ought to invest a bunch in improving that, I'd hope that a lot of > > that work would be largely independent of the language the tests are written > > in. > > We do a lot more acceptance testing than internal testing, which came > up as a major complaint from me and others during the unconference. > One of the reasons people avoid writing internal tests in Perl is > because it's very painful to find a rhythm with Test::More. What definition of internal tests are you using here? I think a lot of our tests are complicated, fragile and slow because we almost exclusively do end-to-end tests, because with a few exceptions we don't have a way to exercise code in a more granular way. > > > When it comes to third-party packages, which I think we're > > > probably going to want in moderation, we would still need to discuss > > > supply chain safety. Python is not as mature here as, say, Go. > > > > What external dependencies are you imagining? > > The OAuth pytest suite makes extensive use of > - psycopg, to easily drive libpq; That's probably not going to fly. It introduces painful circular dependencies between building postgres (for libpq), building psycopg (requiring libpq) and testing postgres (requiring psycopg). You *can* solve such issues, but we've debated that in the past, and I doubt we'll find agreement on the awkwardness it introduces. > - construct, for on-the-wire packet representations and manipulation; and That seems fairly minimal. > - pyca/cryptography, for easy generation of certificates and manual > crypto testing. That's a bit more painful, but I guess maybe not unrealistic? > I'd imagine each would need considerable discussion, if there is > interest in doing the same things that I do with them. One thing worth thinking about is that such dependencies have to work on a relatively large number of platforms / architectures. A lot of projects don't... Greetings, Andres Freund
On Wed, 12 Jun 2024 at 17:50, Andres Freund <andres@anarazel.de> wrote: > > The OAuth pytest suite makes extensive use of > > - psycopg, to easily drive libpq; > > That's probably not going to fly. It introduces painful circular dependencies > between building postgres (for libpq), building psycopg (requiring libpq) and > testing postgres (requiring psycopg). > > You *can* solve such issues, but we've debated that in the past, and I doubt > we'll find agreement on the awkwardness it introduces. psycopg has a few implementations binary, c, & pure python. The pure python one can be linked to a specific libpq.so file at runtime[1]. As long as we don't break the libpq API (which we shouldn't), we can just point it to the libpq compiled by meson/make. We wouldn't be able to use the newest libpq features that way (because psycopg wouldn't know about them), but that seems totally fine for most usages (i.e. sending a query over a connection). If we really want to use those from the python tests we could write our own tiny CFFI layer specifically for those. > One thing worth thinking about is that such dependencies have to work on a > relatively large number of platforms / architectures. A lot of projects > don't... Do they really? A bunch of the Perl tests we just skip on windows or uncommon platforms. I think we could do the same for these.
On Wed, Jun 12, 2024 at 7:08 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote: > On Wed, 12 Jun 2024 at 17:50, Andres Freund <andres@anarazel.de> wrote: > > > The OAuth pytest suite makes extensive use of > > > - psycopg, to easily drive libpq; > > > > That's probably not going to fly. It introduces painful circular dependencies > > between building postgres (for libpq), building psycopg (requiring libpq) and > > testing postgres (requiring psycopg). > > > > You *can* solve such issues, but we've debated that in the past, and I doubt > > we'll find agreement on the awkwardness it introduces. > > psycopg has a few implementations binary, c, & pure python. The pure > python one can be linked to a specific libpq.so file at runtime[1]. As > long as we don't break the libpq API (which we shouldn't), we can just > point it to the libpq compiled by meson/make. We wouldn't be able to > use the newest libpq features that way (because psycopg wouldn't know > about them), but that seems totally fine for most usages (i.e. sending > a query over a connection). If we really want to use those from the > python tests we could write our own tiny CFFI layer specifically for > those. I guess you mean pg8000. Note that pg8000 and psycopg2 have some differences in interpretation of datatypes (AFAIR arrays, jsonb...). So, it would be easier to chose one particular driver. However, with a bit efforts it's possible to make all the code driver agnostic. ------ Regards, Alexander Korotkov Supabase
> On 12 Jun 2024, at 18:08, Jelte Fennema-Nio <postgres@jeltef.nl> wrote: > On Wed, 12 Jun 2024 at 17:50, Andres Freund <andres@anarazel.de> wrote: >> One thing worth thinking about is that such dependencies have to work on a >> relatively large number of platforms / architectures. A lot of projects >> don't... > > Do they really? A bunch of the Perl tests we just skip on windows or > uncommon platforms. I think we could do the same for these. For a project intended to improve on the status quo it seems like a too low bar to just port over the deficincies in thething we’re trying to improve over. ./daniel
On Wed, 12 Jun 2024 at 18:08, Jelte Fennema-Nio <postgres@jeltef.nl> wrote: > > On Wed, 12 Jun 2024 at 17:50, Andres Freund <andres@anarazel.de> wrote: > > > The OAuth pytest suite makes extensive use of > > > - psycopg, to easily drive libpq; > > > > That's probably not going to fly. It introduces painful circular dependencies > > between building postgres (for libpq), building psycopg (requiring libpq) and > > testing postgres (requiring psycopg). > > psycopg has a few implementations binary, c, & pure python. The pure > python one can be linked to a specific libpq.so file at runtime[1]. As This is true, but [citation needed] :D I assume the pointer wanted to be https://www.psycopg.org/psycopg3/docs/api/pq.html#pq-impl I see the following use cases and how I would use psycopg to implement them: - by installing 'psycopg[binary]' you would get a binary bundle shipping with a stable version of the libpq, so you can test the database server regardless of libpq instabilities in the same codebase. - using the pure Python psycopg (enforced by exporting 'PSYCOPG_IMPL=python') you would use the libpq found on the LD_LIBRARY_PATH, which can be useful to test regressions to the libpq itself. - if you want to test new libpq functions you can reach them in Python by dynamic lookup. See [2] for an example of a function only available from libpq v17. [2]: https://github.com/psycopg/psycopg/blob/2bf7783d66ab239a2fa330a842fd461c4bb17c48/psycopg/psycopg/pq/_pq_ctypes.py#L564-L569 -- Daniele
On Wed, Jun 12, 2024 at 7:34 PM Alexander Korotkov <aekorotkov@gmail.com> wrote: > On Wed, Jun 12, 2024 at 7:08 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote: > > On Wed, 12 Jun 2024 at 17:50, Andres Freund <andres@anarazel.de> wrote: > > > > The OAuth pytest suite makes extensive use of > > > > - psycopg, to easily drive libpq; > > > > > > That's probably not going to fly. It introduces painful circular dependencies > > > between building postgres (for libpq), building psycopg (requiring libpq) and > > > testing postgres (requiring psycopg). > > > > > > You *can* solve such issues, but we've debated that in the past, and I doubt > > > we'll find agreement on the awkwardness it introduces. > > > > psycopg has a few implementations binary, c, & pure python. The pure > > python one can be linked to a specific libpq.so file at runtime[1]. As > > long as we don't break the libpq API (which we shouldn't), we can just > > point it to the libpq compiled by meson/make. We wouldn't be able to > > use the newest libpq features that way (because psycopg wouldn't know > > about them), but that seems totally fine for most usages (i.e. sending > > a query over a connection). If we really want to use those from the > > python tests we could write our own tiny CFFI layer specifically for > > those. > > I guess you mean pg8000. Note that pg8000 and psycopg2 have some > differences in interpretation of datatypes (AFAIR arrays, jsonb...). > So, it would be easier to chose one particular driver. However, with > a bit efforts it's possible to make all the code driver agnostic. Ops, this is probably outdated due to presence of psycopg3, as pointed by Daniele Varrazzo [1]. Links. 1. https://www.postgresql.org/message-id/CA%2Bmi_8Zj0gpzPKUEcEx2mPOAsm0zPvznhbcnQDA_eeHVnVqg9Q%40mail.gmail.com ------ Regards, Alexander Korotkov Supabase
> On 12 Jun 2024, at 17:50, Andres Freund <andres@anarazel.de> wrote: > On 2024-06-11 07:28:23 -0700, Jacob Champion wrote: >> The OAuth pytest suite makes extensive use of >> - psycopg, to easily drive libpq; > > That's probably not going to fly. It introduces painful circular dependencies > between building postgres (for libpq), building psycopg (requiring libpq) and > testing postgres (requiring psycopg). I might be missing something obvious, but if we use a third-party libpq driver in the testsuite doesn't that imply that a patch adding net new functionality to libpq also need to add it to the driver in order to write the tests? I'm thinking about the SCRAM situation a few years back when drivers weren't up to date. -- Daniel Gustafsson
On Wed, 12 Jun 2024 at 19:30, Daniel Gustafsson <daniel@yesql.se> wrote: > I might be missing something obvious, but if we use a third-party libpq driver > in the testsuite doesn't that imply that a patch adding net new functionality > to libpq also need to add it to the driver in order to write the tests? I'm > thinking about the SCRAM situation a few years back when drivers weren't up to > date. As Jelte pointed out, new libpq functions can be tested via CFFI. I posted a practical example in a link upthread (pure Python Psycopg is entirely implemented on FFI). -- Daniele
On Wed, Jun 12, 2024 at 1:30 PM Daniel Gustafsson <daniel@yesql.se> wrote: > > On 12 Jun 2024, at 17:50, Andres Freund <andres@anarazel.de> wrote: > > On 2024-06-11 07:28:23 -0700, Jacob Champion wrote: > > >> The OAuth pytest suite makes extensive use of > >> - psycopg, to easily drive libpq; > > > > That's probably not going to fly. It introduces painful circular dependencies > > between building postgres (for libpq), building psycopg (requiring libpq) and > > testing postgres (requiring psycopg). > > I might be missing something obvious, but if we use a third-party libpq driver > in the testsuite doesn't that imply that a patch adding net new functionality > to libpq also need to add it to the driver in order to write the tests? I'm > thinking about the SCRAM situation a few years back when drivers weren't up to > date. Yeah, I don't think depending on psycopg2 is practical at all. We can either shell out to psql like we do now, or we can use something like CFFI. On the overall topic of this thread, I personally find most of the rationale articulated in the original message unconvincing. Many of those improvements could be made on top of the Perl framework we have today, and some of them have been discussed, but nobody's done the work. I also don't understand the argument that assert a == b is some new and wonderful thing; I mean, you can already do is($a,$b,"test name") which *also* shows you the values when they don't match, and includes a test name, too! I personally think that most of the frustration associated with writing TAP tests has to do with (1) Windows behavior being randomly different than on other platforms in ways that are severely under-documented, (2) PostgreSQL::Test::whatever being somewhat clunky and under-designed, and (3) the general difficulty of producing race-free test cases. A new test framework isn't going to solve (3), and (1) and (2) could be fixed anyway. However, I understand that a lot of people would prefer to code in Python than in Perl. I am not one of them: I learned Perl in the early nineties, and I haven't learned Python yet. Nonetheless, Python being more popular than Perl is a reasonable reason to consider allowing its use in PostgreSQL. But if that's the reason, let's be up front about it. I do think we want a scripting language here i.e. not C. -- Robert Haas EDB: http://www.enterprisedb.com
On Tue, Jun 11, 2024 at 4:48 PM Noah Misch <noah@leadboat.com> wrote: > If we're going to test in a non-Perl language, I'd pick C over Python. We already test in C, though? If the complaint is that those tests are driven by Perl, I agree that something like libcheck or GTest or whatever people are using nowadays would be nicer. But that can be added at any point; the activation energy for a new C-based test runner seems pretty small. IMO, there's no reason to pick it _over_ another language, when we already support C tests and agree that developers need to be fluent in C. > We'd need a library like today's Perl > PostgreSQL::Test to make C-language tests nice, but the same would apply to > any new language. I think the effort required in rebuilding end-to-end tests in C is going to be a lot different than in pretty much any other modern high-level language, so I don't agree that "the same would apply". For the five problem statements I put forward, I think C moves the needle for zero of them. It neither solves the problems we have nor gives us stronger tools to solve them ourselves. And for my personally motivating use case of OAuth, where I need to manipulate HTTP and JSON and TLS and so on and so forth, implementing all of that in C would be an absolute nightmare. Given that choice, I would rather use Perl -- and that's saying something, because I like C a lot more than I like Perl -- because it's the difference between being given a rusty but still functional table saw, and being given a box of Legos to build a "better" table saw, when all I want to do is cut a 2x4 in half and move on with my work. I will use the rusty saw if I have to. But I want to get a better saw -- that somebody else with experience in making saws has constructed, and other people are pretty happy with -- as opposed to building my own. > I also want the initial scope to be the new language coexisting with the > existing Perl tests. Strongly agreed. Thanks, --Jacob
On Wed, Jun 12, 2024 at 4:40 AM Jelte Fennema-Nio <postgres@jeltef.nl> wrote: > I think C#, Java, Go, Rust, Kotlin, and Swift would be acceptable > choices for me (and possibly some more). They allow some type of > introspection, they have a garbage collector, and their general > tooling is quite good. > > But I think a dynamically typed scripting language is much more > fitting for writing tests like this. I love static typing for > production code, but imho it really doesn't have much benefit for > tests. +1. I write mostly protocol mocks and glue code in my authn testing, to try to set up the system into some initial state and then break it. Of the languages mentioned here, I've only used C#, Java, and Go. If I had to reimplement my tests, I'd probably reach for Go out of all of those, but the glue would still be more painful than it probably needs to be. > As scripting languages go, the ones that are still fairly heavily in > use are Javascript, Python, Ruby, and PHP. I think all of those could > probably work, but my personal order of preference would be Python, > Ruby, Javascript, PHP. - Python is the easiest language I've personally used to glue things together, bar none. - I like Ruby as a language but have no experience using it for testing. (RSpec did come up during the unconference session and subsequent hallway conversations.) - Javascript is a completely different mental model from what we're used to, IMO. I think we're likely to spend a lot of time fighting the engine unless everyone is very clear on how it works. - I don't see a use case for PHP here. > TO CLARIFY: This thread is not a proposal to replace Perl with Python. > It's a proposal to allow people to also write tests in Python. +1. It doesn't need to replace anything. It just needs to help us do more things than we're currently doing. --Jacob
On Wed, Jun 12, 2024 at 4:48 AM Alexander Korotkov <aekorotkov@gmail.com> wrote: > Generally, testgres was initially designed as Python analogue of what > we have in src/test/perl/PostgreSQL/Test. In particular its > testgres.PostgresNode is analogue of PostgreSQL::Test::Cluster. It > comes under PostgreSQL License. So, I wonder if we could revise it > and fetch most part of it into our source tree. Okay. If there's wide interest in a port of PostgreSQL::Test::Cluster, that might be something to take a look at. (Since I'm focused on testing things that the current Perl suite can't do at all, I would probably not be the first to volunteer.) --Jacob
On Wed, Jun 12, 2024 at 8:50 AM Andres Freund <andres@anarazel.de> wrote: > I think I might have formulated my paragraph above badly - I didn't mean that > we should move away from perl tests tomorrow, but that we need a path forward > that allows folks to write tests without perl. Okay, agreed. > > - Tests aren't cheap, but in my experience, the maintenance-cost math > > for tests is a lot different than the math for implementations. > > At the moment they tend to be *more* expensive often, due to spurious > failures. That's mostly not perl's fault, don't get me wrong, but us not > having better infrastructure for testing complicated behaviour and/or testing > things on a more narrow basis. Well, okay, but I'm not sure how to respond to this in the frame of this discussion. Bad tests will continue to exist. I am trying to add a tool that, in my view, has made it easier for me to test complicated behavior than what we currently have. I can't prove that it will solve other issues too. > > Well, scopes are pretty front and center when you start building > > pytest fixtures, and the complicated longer setups will hopefully > > converge correctly early on and be reused everywhere else. I imagine > > no one wants to build cluster setup from scratch. > > One (the?) prime source of state in our tap tests is the > database. Realistically we can't just tear that one down and reset it between > tests without causing the test times to explode. So we'll have to live with > some persistent state. Yes? If I've given the impression that I disagree, sorry; I agree. > > On a slight tangent, is this not a problem today? > > It is, but that doesn't mean making it even bigger is unproblematic :) Given all that's been said, I don't understand why you think the problem would get bigger. We would cache expensive state that we need, including the cluster, and pytest lets us do that, and my test suite does that. I've never written a suite that spun up a separate cluster for every single test and then immediately threw it away. (If you want to _enable_ that behavior, to test in extreme isolation, then pytest lets you do that too. But it's not something we should do by default.) > > We do a lot more acceptance testing than internal testing, which came > > up as a major complaint from me and others during the unconference. > > One of the reasons people avoid writing internal tests in Perl is > > because it's very painful to find a rhythm with Test::More. > > What definition of internal tests are you using here? There's a spectrum from unit-testing unexported functions all the way to end-to-end acceptance, and personally I believe that anything finer-grained than end-to-end acceptance is unnecessarily painful. My OAuth suite sits somewhere in the middle, where it mocks the protocol layer and can test the client and server as independent pieces. Super useful for OAuth, which is asymmetrical. I'd like to additionally see better support for unit tests of backend internals, but I don't know those seams as well as all of you do and I should not be driving that. I don't think Python will necessarily help you with it. But it sure helped me break apart the client and the server while enjoying the testing process, and other people want to do that too, so that's what I'm pushing for. > I think a lot of our tests are complicated, fragile and slow because we almost > exclusively do end-to-end tests, because with a few exceptions we don't have a > way to exercise code in a more granular way. Yep. > That's probably not going to fly. It introduces painful circular dependencies > between building postgres (for libpq), building psycopg (requiring libpq) and > testing postgres (requiring psycopg). I am trying very hard not to drag that, which I understand is controversial and is in no way a linchpin of my proposal, into the discussion of whether or not we should try supporting pytest. I get it; I understand that the circular dependency is weird; there are alternatives if it's unacceptable; none of that has anything to do with Python+pytest. > One thing worth thinking about is that such dependencies have to work on a > relatively large number of platforms / architectures. A lot of projects > don't... Agreed. Thanks, --Jacob
On Wed, Jun 12, 2024 at 10:30 AM Daniel Gustafsson <daniel@yesql.se> wrote: > I might be missing something obvious, but if we use a third-party libpq driver > in the testsuite doesn't that imply that a patch adding net new functionality > to libpq also need to add it to the driver in order to write the tests? I use the third-party driver to perform the "basics" at a high level -- connections, queries during cluster setup, things that don't involve ABI changes. For new ABI I use ctypes, or as other people have mentioned CFFI would work. --Jacob
On Wed, Jun 12, 2024 at 01:40:30PM +0200, Jelte Fennema-Nio wrote: > On Wed, 12 Jun 2024 at 01:48, Noah Misch <noah@leadboat.com> wrote: > > I also want the initial scope to be the new language coexisting with the > > existing Perl tests. If a bulk translation ever happens, it should happen > > long after the debut of the new framework. That said, I don't much trust a > > human-written bulk language translation to go through without some tests > > accidentally ceasing to test what they test in Perl today. > > I definitely don't think we should rewrite all the tests that we have > in Perl today into some other language. But I do think that whatever > language we choose, that language should make it as least as easy to > write tests, as easy to read them and as easy to see that they are > testing the intended thing, as is currently the case for Perl. > Rewriting a few Perl tests into the new language, even if not merging > the rewrite, is a good way of validating that imho. Agreed. > PS. For PgBouncer I actually hand-rewrote all the tests that we had in > bash (which is the worst testing language ever) in Python and doing so > actually found more bugs in PgBouncer code that our bash tests > wouldn't catch. So it's not necessarily the case that you lose > coverage by rewriting tests. Yep.
Hi, (I don't have an opinion which language should be selected here.) In <CAOYmi+mA7-uNqpY-0jNZY=fE-QsbfeM1j5Mc-vu1Xm+=B8NOXA@mail.gmail.com> "Re: RFC: adding pytest as a supported test framework" on Wed, 12 Jun 2024 12:31:23 -0700, Jacob Champion <jacob.champion@enterprisedb.com> wrote: > - I like Ruby as a language but have no experience using it for > testing. (RSpec did come up during the unconference session and > subsequent hallway conversations.) If we want to select Ruby, I can help. (I'm a Ruby committer and a maintainer of a testing framework bundled in Ruby.) I'm using Ruby for PGroonga's tests that can't be covered by pg_regress. For example, streaming replication related tests. PGroonga has a small utility for it: https://github.com/pgroonga/pgroonga/blob/main/test/helpers/sandbox.rb Here is a streaming replication test with it: https://github.com/pgroonga/pgroonga/blob/main/test/test-streaming-replication.rb I'm using test-unit as testing framework that is bundled in Ruby: https://github.com/test-unit/test-unit/ I don't recommend that we use RSpec as testing framework even if we select Ruby. RSpec may change API. (RSpec did it several times in the past.) If testing framework changes API, we need to rewrite our tests to adapt the change. I'll never change test-unit API because I don't want to rewrite existing tests. Thanks, -- kou
On Wed, 12 Jun 2024 at 18:46, Daniele Varrazzo <daniele.varrazzo@gmail.com> wrote: > This is true, but [citation needed] :D I assume the pointer wanted to > be https://www.psycopg.org/psycopg3/docs/api/pq.html#pq-impl Ugh, yes I definitely meant to add a link to that [1]. I meant this one though: [1]: https://www.psycopg.org/psycopg3/docs/basic/install.html#pure-python-installation > - using the pure Python psycopg (enforced by exporting > 'PSYCOPG_IMPL=python') you would use the libpq found on the > LD_LIBRARY_PATH, which can be useful to test regressions to the libpq > itself. This indeed was the main idea I had in mind. > - if you want to test new libpq functions you can reach them in Python > by dynamic lookup. See [2] for an example of a function only available > from libpq v17. > > [2]: https://github.com/psycopg/psycopg/blob/2bf7783d66ab239a2fa330a842fd461c4bb17c48/psycopg/psycopg/pq/_pq_ctypes.py#L564-L569 Yeah, that dynamic lookup would work. But due to the cyclic dependency on postgres commit vs psycopg PR we couldn't depend on psycopg for those dynamic lookups. So we'd need to have our own dynamic lookup code to do this. I don't see a huge problem with using psycopg for already released commonly used features (i.e. connecting to postgres and doing queries), but still use our own custom dynamic lookup for the rare tests that test newly added features. But I can definitely see people making the argument that if we need to write & maintain some dynamic lookup code ourselves anyway, we might as well have all the dynamic lookup code in our repo to avoid dependencies. On Wed, 12 Jun 2024 at 18:46, Daniele Varrazzo <daniele.varrazzo@gmail.com> wrote: > > On Wed, 12 Jun 2024 at 18:08, Jelte Fennema-Nio <postgres@jeltef.nl> wrote: > > > > On Wed, 12 Jun 2024 at 17:50, Andres Freund <andres@anarazel.de> wrote: > > > > The OAuth pytest suite makes extensive use of > > > > - psycopg, to easily drive libpq; > > > > > > That's probably not going to fly. It introduces painful circular dependencies > > > between building postgres (for libpq), building psycopg (requiring libpq) and > > > testing postgres (requiring psycopg). > > > > psycopg has a few implementations binary, c, & pure python. The pure > > python one can be linked to a specific libpq.so file at runtime[1]. As > > This is true, but [citation needed] :D I assume the pointer wanted to > be https://www.psycopg.org/psycopg3/docs/api/pq.html#pq-impl > > I see the following use cases and how I would use psycopg to implement them: > > - by installing 'psycopg[binary]' you would get a binary bundle > shipping with a stable version of the libpq, so you can test the > database server regardless of libpq instabilities in the same > codebase. > - using the pure Python psycopg (enforced by exporting > 'PSYCOPG_IMPL=python') you would use the libpq found on the > LD_LIBRARY_PATH, which can be useful to test regressions to the libpq > itself. > - if you want to test new libpq functions you can reach them in Python > by dynamic lookup. See [2] for an example of a function only available > from libpq v17. > > [2]: https://github.com/psycopg/psycopg/blob/2bf7783d66ab239a2fa330a842fd461c4bb17c48/psycopg/psycopg/pq/_pq_ctypes.py#L564-L569 > > -- Daniele
On Tue, Jun 11, 2024 at 8:05 AM Andrew Dunstan <andrew@dunslane.net> wrote: > > > On 2024-06-10 Mo 21:49, Andres Freund wrote: > > Hi, > > On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote: > > I'm not sure what part of the testing infrastructure you think is > unmaintained. For example, the last release of Test::Simple was all the way > back on April 25. > > IPC::Run is quite buggy and basically just maintained by Noah these days. > > > Yes, that's true. I think the biggest pain point is possibly the recovery tests. > > Some time ago I did some work on wrapping libpq using the perl FFI module. It worked pretty well, and would mean we couldprobably avoid many uses of IPC::Run, and would probably be substantially more efficient (no fork required). It wouldn'tavoid all uses of IPC::Run, though. > > But my point was mainly that while a new framework might have value, I don't think we need to run out and immediately rewriteseveral hundred TAP tests. Let's pick the major pain points and address those. FWIW, I felt a lot of pain trying to write recovery TAP tests with IPC::Run's pumping functionality. It was especially painful (as someone who knows even less Perl than the "street fighting Perl" Thomas Munro has described having) before the additional test infrastructure was added in BackgroudPsql.pm last year. As an example of the "worst case", it took me two full work days to go from a repro (with psql sessions on a primary and replica node) of the vacuum hang issue being explored in [1] to a sort-of working TAP test which demonstrated it - and that was with help from several other committers. Granted, this is a complex case. A small part of the issue is that, as Tristan has said elsewhere, there aren't good developer tool integrations that I know about for Perl. I use neovim's LSP support for C and Python (in other projects), and there is a whole ecosystem of tools I can use for both C and Python. I know not everyone likes or needs these, but I find that they help me write and debug code faster. I had offered to take a stab at writing some of the BackgroundPsql test infrastructure in Python. I haven't started exploring that yet or looking at what Jacob has done so far, but I am optimistic that this is an area where it is worth seeing what is available to us outside of IPC::Run. - Melanie [1] https://www.postgresql.org/message-id/CAAKRu_bXH2g_pchG7rN_4fs-_6_kVbbJ97gYRoN0Zdb9P04Wag%40mail.gmail.com
On Wed, 12 Jun 2024 at 21:07, Robert Haas <robertmhaas@gmail.com> wrote:
> Yeah, I don't think depending on psycopg2 is practical at all. We can
> either shell out to psql like we do now, or we can use something like
> CFFI.
Quick clarification I meant psycopg3, not psycopg2. And I'd very much
like to avoid using psql for sending queries, luckily CFFI in python
is very good.
> Many of
> those improvements could be made on top of the Perl framework we have
> today, and some of them have been discussed, but nobody's done the
> work.
I agree it's not a technical issue. It is a people issue. There are
very few people skilled in Perl active in the community. And most of
those are very senior hackers that have much more important things to
do that make our Perl testing framework significantly better. And the
less senior people that might see improving tooling as a way to get
help out in the community, are try to stay away from Perl with a 10
foot pole. So the result is, nothing gets improved. Especially since
very few people outside our community improve this tooling either.
I also don't understand the argument that assert a == b is some
> new and wonderful thing; I mean, you can already do is($a,$b,"test
> name") which *also* shows you the values when they don't match, and
> includes a test name, too!
Sure you can, if you know the function exists. And clearly not
everyone knows that it exists, as the quick grep below demonstrates:
❯ grep 'ok(.* == .*' **.pl | wc -l
41
But apart from the obvious syntax doing what you want, the output is
also much better when looking at a slightly more complex case. With
the following code:
def some_returning_func():
return 1234
def some_func(val):
if val > 100:
return 100
return val
def test_mytest():
assert some_func(some_returning_func()) == 50
Pytest will show the following output
def test_mytest():
> assert some_func(some_returning_func()) == 50
E assert 100 == 50
E + where 100 = some_func(1234)
E + where 1234 = some_returning_func()
I have no clue how you could get output that's even close to that
clear with Perl.
Another problem I run into is that, as you probably know, sometimes
you need to look at the postgres logs to find out what actually went
wrong. Currently the only way to find them (for me) is following the
following steps: hmm, let me figure out what that directory was called
again... ah okay it is build/testrun/pg_upgrade/001_basic/... okay
let's start opening log files that all have very similar names until
find the right one.
When a test in pytest fails it automatically outputs all stdout/stderr
that was outputted, and hides it on success. So for the PgBouncer test
suite. I simply send all the relevant log files to stdout, prefixed by
some capitalized identifying line with a few newlines around it.
Something like "PG_LOG: /path/to/actual/logfile". Then when a test
fails in my terminal window I can look at the files related to the
failed test instantly. This allows me to debug failures much faster.
A related thing that also doesn't help at all is that (afaik) seeing
any of the perl tap test output in your terminal requires running
`meson test` with the -v option, and then scrolling up past all the
super verbose output of successfully passing tests to find out what
exactly failed in the single test that failed. And if you don't want
to do that you have to navigate to the magic directory path (
build/testrun/pg_upgrade/001_basic/) of the specific tests to look at
the stdout file there... Which then turns out not to even be there if
you actually had a compilation failure in your perl script (which
happens very often to anyone that doesn't use perl often). So now you
have to scroll up anyway.
Pytest instead is very good at only showing output for the tests that
failed, and hiding pretty much all output for the tests that passed.
> On 13 Jun 2024, at 00:34, Melanie Plageman <melanieplageman@gmail.com> wrote: > FWIW, I felt a lot of pain trying to write recovery TAP tests with > IPC::Run's pumping functionality. It was especially painful (as > someone who knows even less Perl than the "street fighting Perl" > Thomas Munro has described having) before the additional test > infrastructure was added in BackgroudPsql.pm last year. A key aspect of this, which isn't specific to Perl or our use of it, is that this was done in backbranches which doesn't have the (recently) much improved BackgroundPsql.pm. The quality of our tools and the ease of use they provide is directly related to the investment we make into continuously improving our testharness. Regardless of which toolset we adopt, if we don't make this investment (taking learnings from the past years and toolsets into account) we're bound to repeat this thread in a few years advocating for toolset X+1. -- Daniel Gustafsson
On Wed, Jun 12, 2024 at 6:43 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote: > I agree it's not a technical issue. It is a people issue. There are > very few people skilled in Perl active in the community. And most of > those are very senior hackers that have much more important things to > do that make our Perl testing framework significantly better. And the > less senior people that might see improving tooling as a way to get > help out in the community, are try to stay away from Perl with a 10 > foot pole. So the result is, nothing gets improved. Especially since > very few people outside our community improve this tooling either. I agree with you, but I'm skeptical that solving it will be as easy as switching to Python. For whatever reason, it seems like every piece of infrastructure that the PostgreSQL community has suffers from severe neglect. Literally everything I know of either has one or maybe two very senior hackers maintaining it, or no maintainer at all. Andrew maintains the buildfarm and it evolves quite slowly. Andres did all the work on meson, with some help from Peter. Thomas maintains cfbot as a skunkworks. The Perl-based TAP test framework gets barely any love at all. The CommitFest application is pretty much totally stagnant, and in fact is a great example of what I'm talking about here: I wrote an original version in Perl and somebody -- I think Magnus -- rewrote it in a more maintainable framework -- and then the development pace went to basically zero. All of this stuff is critical project infrastructure and yet it feels like nobody wants to work on it. Now, this case may prove to be an exception to that rule and that will be great. But what I think is a lot more likely is that we'll get a lot of pressure to commit something as soon as parity with the Perl TAP test system has been achieved, or maybe even before that, and then the rate of further improvements will slow to a trickle. That's not to say that sticking with Perl is better. A quick Google search finds a web page that says Python is two orders of magnitude more popular than Perl, and that's not something we should just ignore. But I still think it's fair to question whether the preference of many developers for Python over Perl will translate into sustained investment in improving the infrastructure. Again, I will be thrilled if it does, but that just doesn't seem to be the way that things go around here, and I bet the reasons go well beyond choice of programming language. -- Robert Haas EDB: http://www.enterprisedb.com
Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Jun 12, 2024 at 6:43 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
>> I agree it's not a technical issue. It is a people issue. There are
>> very few people skilled in Perl active in the community. And most of
>> those are very senior hackers that have much more important things to
>> do that make our Perl testing framework significantly better. And the
>> less senior people that might see improving tooling as a way to get
>> help out in the community, are try to stay away from Perl with a 10
>> foot pole. So the result is, nothing gets improved. Especially since
>> very few people outside our community improve this tooling either.
> I agree with you, but I'm skeptical that solving it will be as easy as
> switching to Python. For whatever reason, it seems like every piece of
> infrastructure that the PostgreSQL community has suffers from severe
> neglect.
Yeah. In this case it's perhaps more useful to look at our external
dependencies, the large majority of which are suffering from age
and neglect:
* autoconf & gmake (although meson may get us out from under these)
* bison
* flex
* perl
* tcl
* regex library (basically from tcl)
* libxml2
* kerberos
* ldap
* pam
* uuid library
I think the basic problem is inherent in being a successful long-lived
project. Or maybe we're just spectacularly bad at picking which
things to depend on. Whichever it is, we'd better have a 10- or 20-
year perspective when thinking about adopting new major dependencies.
In the case at hand, I share Robert's doubts about Python. Sure it's
more popular than Perl, but I don't think it's actually better, and
in some ways it's worse. (The moving-target package collection was
mentioned as a problem, for instance.) Is it going to age better
than Perl? Doubt it.
I wonder if we should be checking out some of the other newer
languages that were mentioned upthread. It feels like going to
Python here will lead to having two testing infrastructures with
mas-o-menos the same capabilities, leaving us with a situation
where people have to know both languages in order to make sense of
our test suite. I find it hard to picture that as an improvement
over the status quo.
regards, tom lane
On Thu, 13 Jun 2024 at 15:38, Robert Haas <robertmhaas@gmail.com> wrote: > For whatever reason, it seems like every piece of > infrastructure that the PostgreSQL community has suffers from severe > neglect. Literally everything I know of either has one or maybe two > very senior hackers maintaining it, or no maintainer at all. Andrew > maintains the buildfarm and it evolves quite slowly. Andres did all > the work on meson, with some help from Peter. Thomas maintains cfbot > as a skunkworks. The Perl-based TAP test framework gets barely any > love at all. The CommitFest application is pretty much totally > stagnant, and in fact is a great example of what I'm talking about > here: I wrote an original version in Perl and somebody -- I think > Magnus -- rewrote it in a more maintainable framework -- and then the > development pace went to basically zero. All of this stuff is critical > project infrastructure and yet it feels like nobody wants to work on > it. Overall, I agree with the sentiment of us not maintaining our tooling well (although I think meson maintenance has been pretty decent so far). I think there's a bunch of reasons for this (not all apply to each of the tools): 1. pretty much solely maintained by senior community members who don't have time to maintain it 2. no clear way to contribute. e.g. where should I send a patch/PR for the commitfest app, or the cfbot? 3. (related to 1) unresponsive when somehow contributions are actually sent in (I have two open PRs on the cfbot app from 3 years ago without any response) I think 1 & 3 could be addressed by more easily giving commit/merge access to these tools than to the main PG repo. And I think 2 could be addressed by writing on the relevant wiki page where to go, and probably putting a link to the wiki page on the actual website of the tool. But Perl is at the next level of unmaintained infrastructure. It is actually clear how you can contribute to it, but still no new community members actually want to contribute to it. Also, it's not only unmaintained by us but it's also pretty much unmaintained by the upstream community. > But I still > think it's fair to question whether the preference of many developers > for Python over Perl will translate into sustained investment in > improving the infrastructure. Again, I will be thrilled if it does, > but that just doesn't seem to be the way that things go around here, > and I bet the reasons go well beyond choice of programming language. As you said, no one in our community wants to maintain our testsuite full time. But our test suite consists partially of upstream dependencies and partially of our own code. Right now pretty much no-one improves the ustream code, and pretty much no-one improves our own code. Using a more modern language gives up much more frequent upstream improvements for free, and it will allow new community members to contribute to our own test suite. And I understand you are sceptical that people will contribute to our own test suite, just because it's Python. But as a counterpoint: people are currently already doing exactly that, just outside of the core postgres repo[1][2][3]. I don't see why those people would suddenly stop doing that if we include such a suite in the official repo. Apparently many people hate writing tests in Perl so much that they'd rather build Python test frameworks to test their extensions, than to use/improve the Perl testing framework included in Postgres. [1]: https://github.com/pgbouncer/pgbouncer/tree/master/test [2]: https://github.com/jchampio/pg-pytest-suite [3]: https://github.com/postgrespro/testgres PS. I don't think it makes sense to host our tooling like the commitfest app on our own git server instead of github/gitlab. That only makes it harder for community members to contribute and also much harder to set up CI. I understand the reasons why we use mailing lists for the development of core postgres, but I don't think those apply nearly as much to our tooling repos. And honestly also not to stuff like the website.
On Thu, 13 Jun 2024 at 17:19, Tom Lane <tgl@sss.pgh.pa.us> wrote: > I wonder if we should be checking out some of the other newer > languages that were mentioned upthread. If this is actually something that we want to seriously evaluate, I think that's a significant effort. And I think the people that want a language would need to step in to make that effort. So far Jacob[1], Alexander[2] and me[3] seem to be doing that for Python, and Sutou has done that for Ruby[4]. [1]: https://github.com/pgbouncer/pgbouncer/tree/master/test [2]: https://github.com/jchampio/pg-pytest-suite [3]: https://github.com/postgrespro/testgres [4]: https://github.com/pgroonga/pgroonga/blob/main/test/test-streaming-replication.rb > It feels like going to > Python here will lead to having two testing infrastructures with > mas-o-menos the same capabilities, leaving us with a situation > where people have to know both languages in order to make sense of > our test suite. I find it hard to picture that as an improvement > over the status quo. You don't have to be fluent in writing Python to be able to read and understand tests written in it. As someone familiar with Python I can definitely read our test suite, and I expect everyone smart enough to be fluent in Perl to be able to read and understand Python with fairly little effort too. I think having significantly more tests being written, and those tests being written faster and more correctly, is definitely worth the slight mental effort of learning to read two very similarly looking scripting languages (they both pretty much looking like pseudo code).
Jelte Fennema-Nio <postgres@jeltef.nl> writes:
> You don't have to be fluent in writing Python to be able to read and
> understand tests written in it.
[ shrug... ] I think the same can be said of Perl, with about as
much basis. It matters a lot if you have previous experience with
the language.
regards, tom lane
On Thu, Jun 13, 2024 at 11:19 AM Tom Lane <tgl@sss.pgh.pa.us> wrote: > I wonder if we should be checking out some of the other newer > languages that were mentioned upthread. It feels like going to > Python here will lead to having two testing infrastructures with > mas-o-menos the same capabilities, leaving us with a situation > where people have to know both languages in order to make sense of > our test suite. I find it hard to picture that as an improvement > over the status quo. As I see it, one big problem is that if you pick a language that's too new, it's more likely to fade away. Python is very well-established, e.g. see https://www.tiobe.com/tiobe-index/ That gives Python a rating of 15.39%; vs. Perl at 0.69%. There are other things that you could pick, for sure, like Javascript, but if you want a scripting language that's popular now, Python is hard to beat. And that means it's more likely to still have some life in it 10 or 20 years from now than many other things. Not all sites agree on which programming languages are actually the most popular and I'm not strongly against considering other possibilities, but Python seems to be pretty high on most lists, often #1, and that does matter. -- Robert Haas EDB: http://www.enterprisedb.com
On Thu, Jun 13, 2024 at 7:27 AM Daniel Gustafsson <daniel@yesql.se> wrote: > > > On 13 Jun 2024, at 00:34, Melanie Plageman <melanieplageman@gmail.com> wrote: > > > FWIW, I felt a lot of pain trying to write recovery TAP tests with > > IPC::Run's pumping functionality. It was especially painful (as > > someone who knows even less Perl than the "street fighting Perl" > > Thomas Munro has described having) before the additional test > > infrastructure was added in BackgroudPsql.pm last year. > > A key aspect of this, which isn't specific to Perl or our use of it, is that > this was done in backbranches which doesn't have the (recently) much improved > BackgroundPsql.pm. The quality of our tools and the ease of use they provide > is directly related to the investment we make into continuously improving our > testharness. Regardless of which toolset we adopt, if we don't make this > investment (taking learnings from the past years and toolsets into account) > we're bound to repeat this thread in a few years advocating for toolset X+1. True. And thank you for committing BackgroundPsql.pm (and Andres for starting that work). My specific case is likely one of a poor work person blaming her tools :) - Melanie
On Thu, Jun 13, 2024 at 1:08 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote: > I think 1 & 3 could be addressed by more easily giving commit/merge > access to these tools than to the main PG repo. And I think 2 could be > addressed by writing on the relevant wiki page where to go, and > probably putting a link to the wiki page on the actual website of the > tool. +1. > But Perl is at the next level of unmaintained infrastructure. It is > actually clear how you can contribute to it, but still no new > community members actually want to contribute to it. Also, it's not > only unmaintained by us but it's also pretty much unmaintained by the > upstream community. I feel like I already agreed to this in a previous email and you're continuing to argue with me as if I were disagreeing. > As you said, no one in our community wants to maintain our testsuite > full time. But our test suite consists partially of upstream > dependencies and partially of our own code. Right now pretty much > no-one improves the ustream code, and pretty much no-one improves our > own code. Using a more modern language gives up much more frequent > upstream improvements for free, and it will allow new community > members to contribute to our own test suite. I also agree with this. I'm just not super optimistic about how much of that will actually happen. And I'd like to hear you acknowledge that concern and think about whether it can be addressed in some way, instead of just repeating that we should do it anyway. Because I agree we probably should do it anyway, but that doesn't mean I wouldn't like to see the downsides mitigated as much as we can. In particular, if the proposal is exactly "let's add the smallest possible patch that enables people to write tests in Python and then add a few new tests in Python while leaving almost everything else in Perl, with no migration plan and no clear vision of how the Python support ever gets any better than the minimum stub that is proposed for initial commit," then I don't know that I can vote for that plan. Honestly, that sounds like very little work for the person proposing that minimal patch and a whole lot of work for the rest of the community later on, and the evidence is not in favor of volunteers showing up to take care of that work. The plan should be more front-loaded than that: enough initial development should get done by the people making the proposal that if the work stops after, we don't have another big mess on our hands. Or so I think, anyway. -- Robert Haas EDB: http://www.enterprisedb.com
> On 13 Jun 2024, at 20:09, Melanie Plageman <melanieplageman@gmail.com> wrote: > > On Thu, Jun 13, 2024 at 7:27 AM Daniel Gustafsson <daniel@yesql.se> wrote: >> >>> On 13 Jun 2024, at 00:34, Melanie Plageman <melanieplageman@gmail.com> wrote: >> >>> FWIW, I felt a lot of pain trying to write recovery TAP tests with >>> IPC::Run's pumping functionality. It was especially painful (as >>> someone who knows even less Perl than the "street fighting Perl" >>> Thomas Munro has described having) before the additional test >>> infrastructure was added in BackgroudPsql.pm last year. >> >> A key aspect of this, which isn't specific to Perl or our use of it, is that >> this was done in backbranches which doesn't have the (recently) much improved >> BackgroundPsql.pm. The quality of our tools and the ease of use they provide >> is directly related to the investment we make into continuously improving our >> testharness. Regardless of which toolset we adopt, if we don't make this >> investment (taking learnings from the past years and toolsets into account) >> we're bound to repeat this thread in a few years advocating for toolset X+1. > > True. And thank you for committing BackgroundPsql.pm (and Andres for > starting that work). My specific case is likely one of a poor work > person blaming her tools :) I don't think it is since the tools we had then were really hard to use. I wrote very similar tests to yours for the online checksums patch and they were quite complicated to get right. The point is that the complexity was greatly reduced by the community, and that kind of work will be equally important regardless of toolset. -- Daniel Gustafsson
On Thu, 13 Jun 2024 at 20:11, Robert Haas <robertmhaas@gmail.com> wrote: > > But Perl is at the next level of unmaintained infrastructure. It is > > actually clear how you can contribute to it, but still no new > > community members actually want to contribute to it. Also, it's not > > only unmaintained by us but it's also pretty much unmaintained by the > > upstream community. > > I feel like I already agreed to this in a previous email and you're > continuing to argue with me as if I were disagreeing. Sorry about that. > I also agree with this. I'm just not super optimistic about how much > of that will actually happen. And I'd like to hear you acknowledge > that concern and think about whether it can be addressed in some way, > instead of just repeating that we should do it anyway. Because I agree > we probably should do it anyway, but that doesn't mean I wouldn't like > to see the downsides mitigated as much as we can. I'm significantly more optimistic than you, but I also definitely understand and agree with the concern. I also agree that mitigating that concern beforehand would be a good thing. > In particular, if > the proposal is exactly "let's add the smallest possible patch that > enables people to write tests in Python and then add a few new tests > in Python while leaving almost everything else in Perl, with no > migration plan and no clear vision of how the Python support ever gets > any better than the minimum stub that is proposed for initial commit," > then I don't know that I can vote for that plan. Honestly, that sounds > like very little work for the person proposing that minimal patch and > a whole lot of work for the rest of the community later on, and the > evidence is not in favor of volunteers showing up to take care of that > work. The plan should be more front-loaded than that: enough initial > development should get done by the people making the proposal that if > the work stops after, we don't have another big mess on our hands. > > Or so I think, anyway. I understand and agree with your final stated goal of not ending up in another big mess. It's also clear to me that you don't think the current proposal achieves that goal. So I assume you have some additional ideas for the proposal to help achieve that goal and/or some specific worries that you'd like to get addressed better in the proposal. But currently it's not really clear to me what either of those are. Could you clarify?
On Thu, Jun 13, 2024 at 9:38 AM Robert Haas <robertmhaas@gmail.com> wrote:
I agree with you, but I'm skeptical that solving it will be as easy as
switching to Python. For whatever reason, it seems like every piece of
infrastructure that the PostgreSQL community has suffers from severe
neglect. Literally everything I know of either has one or maybe two
very senior hackers maintaining it, or no maintainer at all.
...
All of this stuff is critical project infrastructure and yet it feels like nobody wants to work on
it.
I feel at least some of this is a visibility / marketing problem. I've not seen any dire requests for help come across on the lists, nor things on the various todos/road maps/ blog posts people make from time to time. If I had, I would have jumped in. And for the record, I'm very proficient with Perl.
Cheers,
Greg
On Thu, Jun 13, 2024 at 2:52 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
> I understand and agree with your final stated goal of not ending up in
> another big mess. It's also clear to me that you don't think the
> current proposal achieves that goal. So I assume you have some
> additional ideas for the proposal to help achieve that goal and/or
> some specific worries that you'd like to get addressed better in the
> proposal. But currently it's not really clear to me what either of
> those are. Could you clarify?
Hmm, I don't know that I have what you're hoping I have, or at least
not any more than what I've said already.
I interpreted Jacob's original email as articulating a goal ("For the
v18 cycle, I would like to try to get pytest [1] in as a supported
test driver, in addition to the current offerings") rather than a
plan. There's no patch set yet and, as I understand it, no detailed
plan for a patch set: that email seemed to focus on the question of
desirability, rather than on outlining a plan of work, which I assume
is still to come. Some things I'd like to see when a patch set does
show up are:
- good documentation for people who have no previous experience with
Python and/or pytest e.g. here's how to set up your environment on
Linux, Windows, macOS, *BSD so you can run the tests, here's how to
run the tests, here's how it's different from the Perl framework we
have now
- no external dependencies on PostgreSQL connectors. psql or libpq
foreign function interface. the latter would be a cool increment of
progress over the status quo.
- at least as much in-tree support for writing tests as we have today
with PostgreSQL::Test::whatever, but not necessarily a 1:1 reinvention
of the stuff we have now, and documentation of those facilities that
is as good or, ideally, better than what we have today.
- high overall code quality and level of maturity, not just something
someone threw together for parity with the Perl system.
- enough tests written for or converted to the new system to give
reviewers confidence that it's truly usable and fit for purpose.
The important thing to me here (as it so often is) is to think like a
maintainer. Imagine that immediately after the patches for this
feature are committed, the developers who did the work all disappear
from the community and are never heard from again. How much pain does
that end us causing? The answer doesn't need to be zero; that is
unrealistic. But it also shouldn't be "well, if that happens we're
going to have to rip the feature out" or "well, a bunch of committers
who didn't want to write tests in Python in the first place are now
going to have to do a lot of work in Python to stabilize the work
already committed."
--
Robert Haas
EDB: http://www.enterprisedb.com
On Thu, Jun 13, 2024 at 3:17 PM Greg Sabino Mullane <htamfids@gmail.com> wrote: > I feel at least some of this is a visibility / marketing problem. I've not seen any dire requests for help come acrosson the lists, nor things on the various todos/road maps/ blog posts people make from time to time. If I had, I wouldhave jumped in. And for the record, I'm very proficient with Perl. I agree with all of that! -- Robert Haas EDB: http://www.enterprisedb.com
On Thu, Jun 13, 2024 at 11:11 AM Robert Haas <robertmhaas@gmail.com> wrote: > I feel like I already agreed to this in a previous email and you're > continuing to argue with me as if I were disagreeing. I also think that maybe arguments are starting to sail past each other, and the temperature is starting to climb. (And Jelte may be arguing to all readers of the thread, rather than just a single individual. It's hard to tell with the list format.) And now I see that there's another email that came in while I was writing this, but I think I'm going to have to send this as-is because I can't write emails that fast. > I also agree with this. I'm just not super optimistic about how much > of that will actually happen. And I'd like to hear you acknowledge > that concern and think about whether it can be addressed in some way, > instead of just repeating that we should do it anyway. Because I agree > we probably should do it anyway, but that doesn't mean I wouldn't like > to see the downsides mitigated as much as we can. Okay, +1. > In particular, if > the proposal is exactly "let's add the smallest possible patch that > enables people to write tests in Python and then add a few new tests > in Python while leaving almost everything else in Perl, with no > migration plan and no clear vision of how the Python support ever gets > any better than the minimum stub that is proposed for initial commit," > then I don't know that I can vote for that plan. (that's not the proposal and I know/think you know that but having my original email twisted into that is making me feel a bit crispy) I do not want to migrate, and I have stated so multiple times, which is why I have not proposed a migration plan. Other committers have already expressed resistance to the idea that we would rewrite the Perl stuff. I think they're right. I think we should not. I think we should accept the cost of both Perl and something else for the near-to-medium future, as long as the "something else" gives us value offsetting the additional cost. > Honestly, that sounds > like very little work for the person proposing that minimal patch and > a whole lot of work for the rest of the community later on, and the > evidence is not in favor of volunteers showing up to take care of that > work. Okay, cool. Here: as the person who is 100% signing himself up to do that, time for me to jump in. I have an entire 6000-something-line suite of protocol tests that has been linked like four times above. It does something fundamentally different from the end-to-end Perl suite; it is not a port. It is far from perfect and I do not want to pressure people to adopt it as-is, which is why I have not. In this thread, I am offering it solely as evidence that I have follow-up intent. But I might get hit by a bus. Or, as far as anyone except me knows, I might lose interest after things get hard, which would be sad. Which is why my very first proposal was to add an entry point that can be reverted. The suite is not going to infect the codebase any more than the Perl does. A revert will involve pulling the Meson test entry code, and deleting all pytest subdirectories (of which there is only one, IIRC, in my OAuth suite). > The plan should be more front-loaded than that: enough initial > development should get done by the people making the proposal that if > the work stops after, we don't have another big mess on our hands. For me personally, the problem is the opposite. I have done _so much_ initial development by myself that there's no way it could ever be reviewed and accepted. But I had to do that to get meaningful development done in my style of work, which is focused on security and testability and verifiable implementation. I am trying to carve off pieces of that and say "hey, does this look nice to anyone else?" That will take time, probably over multiple different threads. In the meantime, I don't want to be a serialization point for other people who are excited about trying new testing methods, because very few people are currently doing the exact kind of work I am doing. They may want to do other things, as evidenced by the thread contents. At least one committer would have to sign up to be a serialization point, unfortunately, but I think that's going to be true regardless of plan, if we want multiple non-committer members of the community to be involved instead of just one torch-bearer. Because of how many moving parts and competing interests and personal disagreements there are, I am firmly in the camp of "try something that many people think *might* work better, that can be undone if it sucks, and iterate on it." I want to build community momentum, because I think that's a pretty effective way to change the cultural norms that you're saying you're frustrated with. That doesn't mean I want to do this without a plan; it just means that the plan can involve saying "this is not working and we can undo it" which makes the uncertainty easier to take. --Jacob
On Thu, Jun 13, 2024 at 3:28 PM Jacob Champion <jacob.champion@enterprisedb.com> wrote: > (that's not the proposal and I know/think you know that but having my > original email twisted into that is making me feel a bit crispy) I definitely did not mean to imply that. I took your original email as a goal, rather than a proposal or plan. My statement was strictly intended as a hypothetical because I didn't think any plan had been proposed - I only meant to say that *if* the plan were to do X, that would be a hard sell for me. > I do not want to migrate, and I have stated so multiple times, which > is why I have not proposed a migration plan. Other committers have > already expressed resistance to the idea that we would rewrite the > Perl stuff. I think they're right. I think we should not. I think we > should accept the cost of both Perl and something else for the > near-to-medium future, as long as the "something else" gives us value > offsetting the additional cost. I agree. It's not terribly pretty, IMHO, but it's hard to see doing things any other way. > For me personally, the problem is the opposite. I have done _so much_ > initial development by myself that there's no way it could ever be > reviewed and accepted. But I had to do that to get meaningful > development done in my style of work, which is focused on security and > testability and verifiable implementation. I admire this attitude. I think a lot of people who go off and do a ton of initial work outside core show up and are like "ok, now take all of my code." As you say, that's not realistic. One caveat here, perhaps, is that the focus of the work you've done up until now and the things that I and other community members may want as a condition of merging stuff may be somewhat distinct. You will have naturally been focused on your goals rather than other people's goals, or so I assume. > I am trying to carve off pieces of that and say "hey, does this look > nice to anyone else?" That will take time, probably over multiple > different threads. This makes sense, but I would be a bit wary of splitting it up over too many different threads. It may well make sense to split it up, but it will probably be easier to review if the core work to enable this is one patch set on one thread where someone can read just that one thread and understand the situation, rather than many threads where you have to read them all. > Because of how many moving parts and competing interests and personal > disagreements there are, I am firmly in the camp of "try something > that many people think *might* work better, that can be undone if it > sucks, and iterate on it." I want to build community momentum, because > I think that's a pretty effective way to change the cultural norms > that you're saying you're frustrated with. That doesn't mean I want to > do this without a plan; it just means that the plan can involve saying > "this is not working and we can undo it" which makes the uncertainty > easier to take. As a community, we're really bad at this. Once something gets committed, getting a consensus to revert it is really hard, especially if a major release has happened meanwhile, but most of the time even if it hasn't. It might be a little easier in this case, since after all it's not a directly user-visible feature. But historically what happens if somebody says "hey, there are six unfixed problems with this feature!" is that everybody says "well, you're free to fix the problems if you want, but you're not allowed to revert the feature." And that is *exactly* how we end up with stuff like the current TAP test framework: ripping that out would mean removing all the TAP tests that depend on it, and that wouldn't have achieved consensus two months after the feature went in, let alone today. Now, it has been suggested to me by at least one other person involved with the project that we need to be more open to the kind of thing that you propose here: add experimental things and take them out if it doesn't work out. I can definitely understand that this might be a culturally better approach than what we currently do. So maybe that's the way forward, but it is hard (at least for me) to get past the fear of being the one left holding the bag, and I suspect that other committers have similar fears. What exactly we should do about that, I'm not sure. -- Robert Haas EDB: http://www.enterprisedb.com
On 2024-06-12 We 11:28, Andres Freund wrote:
> Hi,
>
> On 2024-06-11 08:04:57 -0400, Andrew Dunstan wrote:
>> Some time ago I did some work on wrapping libpq using the perl FFI module.
>> It worked pretty well, and would mean we could probably avoid many uses of
>> IPC::Run, and would probably be substantially more efficient (no fork
>> required). It wouldn't avoid all uses of IPC::Run, though.
> FWIW, I'd *love* to see work on this continue. The reduction in test runtime
> on windows is substantial and would shorten the hack->CI->fail->hack loop a
> good bit shorter. And save money.
OK, I will put it high on my list. I just did some checking and it seems
to be feasible on Windows. StrawberryPerl at least has FFI::Platypus out
of the box, so we would not need to turn any great handsprings to make
progress on this on a fairly wide variety of platforms.
What seems a good place to start would be a simple
PostgreSQL::Test::Session object that would allow us to get rid of a
whole heap of start/pump_until/kill cycles and deal with the backend in
a much more straightforward and comprehensible way, not to mention the
possibility of removing lots of $node->{safe_}psql calls.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
On Thu, Jun 13, 2024 at 12:20 PM Robert Haas <robertmhaas@gmail.com> wrote:
> I interpreted Jacob's original email as articulating a goal ("For the
> v18 cycle, I would like to try to get pytest [1] in as a supported
> test driver, in addition to the current offerings") rather than a
> plan.
That's the first part of it.
> There's no patch set yet and, as I understand it, no detailed
> plan for a patch set: that email seemed to focus on the question of
> desirability, rather than on outlining a plan of work, which I assume
> is still to come.
There was a four-step plan sketch at the end of that email, titled "A
Plan". That was not intended to be "the final detailed plan", because
I was soliciting feedback on the exact pieces people wanted to try to
implement first, and I am not the God Emperor of Pytest. But it was
definitely A Plan.
> Some things I'd like to see when a patch set does
> show up are:
>
> - good documentation for people who have no previous experience with
> Python and/or pytest e.g. here's how to set up your environment on
> Linux, Windows, macOS, *BSD so you can run the tests, here's how to
> run the tests, here's how it's different from the Perl framework we
> have now
+1
> - no external dependencies on PostgreSQL connectors. psql or libpq
> foreign function interface. the latter would be a cool increment of
> progress over the status quo.
If this is a -1 for psycopg, then I will cast my vote for ctypes/CFFI
and against psql.
> - at least as much in-tree support for writing tests as we have today
> with PostgreSQL::Test::whatever, but not necessarily a 1:1 reinvention
> of the stuff we have now, and documentation of those facilities that
> is as good or, ideally, better than what we have today.
I think this is way too much expectation for a v1 patch. If you were
committing this by yourself, would you agree to develop the entirety
of PostgreSQL::Test in a single commit, without the benefit of the
buildfarm checking you as you went, and other people trying to write
tests with it?
> - high overall code quality and level of maturity, not just something
> someone threw together for parity with the Perl system.
+1
> - enough tests written for or converted to the new system to give
> reviewers confidence that it's truly usable and fit for purpose.
This is that "know everything up front" tax that I think is not
reasonable for a test framework. If the thing you're trying to avoid
is the foot-in-the-door phenomenon, I would agree with you for a
Postgres feature. But these are tests; we don't ship them, we have
different rules for backporting them, they are developed in a very
different way.
> The important thing to me here (as it so often is) is to think like a
> maintainer. Imagine that immediately after the patches for this
> feature are committed, the developers who did the work all disappear
> from the community and are never heard from again. How much pain does
> that end us causing? The answer doesn't need to be zero; that is
> unrealistic. But it also shouldn't be "well, if that happens we're
> going to have to rip the feature out"
Can you elaborate on why that's not an okay outcome?
> or "well, a bunch of committers
> who didn't want to write tests in Python in the first place are now
> going to have to do a lot of work in Python to stabilize the work
> already committed."
Is it that? If the problem is that, we should address that. Because if
that is truly the fear, I cannot assuage that fear without showing you
something, and I cannot show you something you do not want to see, if
you don't want to write tests in Python in the first place.
--Jacob
On 2024-06-12 We 18:34, Melanie Plageman wrote: > On Tue, Jun 11, 2024 at 8:05 AM Andrew Dunstan <andrew@dunslane.net> wrote: >> >> On 2024-06-10 Mo 21:49, Andres Freund wrote: >> >> Hi, >> >> On 2024-06-10 16:46:56 -0400, Andrew Dunstan wrote: >> >> I'm not sure what part of the testing infrastructure you think is >> unmaintained. For example, the last release of Test::Simple was all the way >> back on April 25. >> >> IPC::Run is quite buggy and basically just maintained by Noah these days. >> >> >> Yes, that's true. I think the biggest pain point is possibly the recovery tests. >> >> Some time ago I did some work on wrapping libpq using the perl FFI module. It worked pretty well, and would mean we couldprobably avoid many uses of IPC::Run, and would probably be substantially more efficient (no fork required). It wouldn'tavoid all uses of IPC::Run, though. >> >> But my point was mainly that while a new framework might have value, I don't think we need to run out and immediatelyrewrite several hundred TAP tests. Let's pick the major pain points and address those. > FWIW, I felt a lot of pain trying to write recovery TAP tests with > IPC::Run's pumping functionality. It was especially painful (as > someone who knows even less Perl than the "street fighting Perl" > Thomas Munro has described having) before the additional test > infrastructure was added in BackgroudPsql.pm last year. As an example > of the "worst case", it took me two full work days to go from a repro > (with psql sessions on a primary and replica node) of the vacuum hang > issue being explored in [1] to a sort-of working TAP test which > demonstrated it - and that was with help from several other > committers. Granted, this is a complex case. The pump stuff is probably the least comprehensible and most fragile part of the whole infrastructure. As I just mentioned to Andres, I'm hoping to make a substantial improvement in that area. > > A small part of the issue is that, as Tristan has said elsewhere, > there aren't good developer tool integrations that I know about for > Perl. I use neovim's LSP support for C and Python (in other projects), > and there is a whole ecosystem of tools I can use for both C and > Python. I know not everyone likes or needs these, but I find that they > help me write and debug code faster. You might find this useful: <https://climatechangechat.com/setting_up_lsp_nvim-lspconfig_and_perl_in_neovim.html> (I don't use neovim - I'm an old emacs dinosaur.) > > I had offered to take a stab at writing some of the BackgroundPsql > test infrastructure in Python. I haven't started exploring that yet or > looking at what Jacob has done so far, but I am optimistic that this > is an area where it is worth seeing what is available to us outside of > IPC::Run. Yeah, like I said, I'm working on reducing our reliance on especially the more fragile parts of IPC::Run. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
On Thu, Jun 13, 2024 at 4:06 PM Jacob Champion <jacob.champion@enterprisedb.com> wrote: > There was a four-step plan sketch at the end of that email, titled "A > Plan". That was not intended to be "the final detailed plan", because > I was soliciting feedback on the exact pieces people wanted to try to > implement first, and I am not the God Emperor of Pytest. But it was > definitely A Plan. Well, OK, now I feel a bit dumb. I guess I missed that or forgot about it. > > - at least as much in-tree support for writing tests as we have today > > with PostgreSQL::Test::whatever, but not necessarily a 1:1 reinvention > > of the stuff we have now, and documentation of those facilities that > > is as good or, ideally, better than what we have today. > > I think this is way too much expectation for a v1 patch. If you were > committing this by yourself, would you agree to develop the entirety > of PostgreSQL::Test in a single commit, without the benefit of the > buildfarm checking you as you went, and other people trying to write > tests with it? Eh... I'm confused. PostgreSQL::Test::Cluster is more than half of the code in that directory, and without it you wouldn't be able to write most of the TAP tests that we have today. You would really want to call this project done without having an equivalent? > > The important thing to me here (as it so often is) is to think like a > > maintainer. Imagine that immediately after the patches for this > > feature are committed, the developers who did the work all disappear > > from the community and are never heard from again. How much pain does > > that end us causing? The answer doesn't need to be zero; that is > > unrealistic. But it also shouldn't be "well, if that happens we're > > going to have to rip the feature out" > > Can you elaborate on why that's not an okay outcome? Well, you just argued that it should be an okay outcome, and I do sort of see your point, but I refer you to my earlier reply about the difficulty of getting anything reverted in the culture as it stands. > > or "well, a bunch of committers > > who didn't want to write tests in Python in the first place are now > > going to have to do a lot of work in Python to stabilize the work > > already committed." > > Is it that? If the problem is that, we should address that. Because if > that is truly the fear, I cannot assuage that fear without showing you > something, and I cannot show you something you do not want to see, if > you don't want to write tests in Python in the first place. I have zero desire to write tests in Python. If I could convince everyone here to spend their time and energy improving the stuff we have in Perl instead of introducing a whole new test framework, I would 100% do that. But I'm pretty sure that I can't, and I think the project needs to pick from among realistic options rather than theoretical ones. Said differently, it's not all about me. -- Robert Haas EDB: http://www.enterprisedb.com
On 2024-06-12 We 11:50, Andres Freund wrote:
Hi, On 2024-06-11 07:28:23 -0700, Jacob Champion wrote:On Mon, Jun 10, 2024 at 1:04 PM Andres Freund <andres@anarazel.de> wrote:Just for context for the rest the email: I think we desperately need to move off perl for tests. The infrastructure around our testing is basically unmaintained and just about nobody that started doing dev stuff in the last 10 years learned perl.Okay. Personally, I'm going to try to stay out of discussions around subtracting Perl and focus on adding Python, for a bunch of different reasons:I think I might have formulated my paragraph above badly - I didn't mean that we should move away from perl tests tomorrow,
OK, glad we're on the same page there. Let's move on.
but that we need a path forward that allows folks to write tests without perl.
OK, although to be honest I'm more interested in fixing some of the things that have made testing with perl a pain, especially the IPC::Run pump stuff.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On 2024-06-13 Th 15:16, Greg Sabino Mullane wrote:
I'm very proficient with Perl.
Yes you are, and just as well!
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On 2024-06-12 We 18:43, Jelte Fennema-Nio wrote: > > I agree it's not a technical issue. It is a people issue. There are > very few people skilled in Perl active in the community. And most of > those are very senior hackers that have much more important things to > do that make our Perl testing framework significantly better. And the > less senior people that might see improving tooling as a way to get > help out in the community, are try to stay away from Perl with a 10 > foot pole. So the result is, nothing gets improved. Especially since > very few people outside our community improve this tooling either. > > FTR, I have put a lot of effort into maintaining and improving the infrastructure over the years. And I don't think there is anything much more important. So I'm going to put more effort in. And I'm not alone. Andres, Alvaro, Noah and Thomas are some of those who have spent a lot of effort on extending and improving our testing. People tend to get a bit hung up about languages. I lost count of the various languages I had learned when it got somewhere north of 30. Still, I understand that perl has a few oddities that make people scratch their heads (as do most languages). It's probably losing market share, along with some of the other things we rely on. Not sure that alone is a reason to move away from it. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
On Thu, Jun 13, 2024 at 4:54 PM Andrew Dunstan <andrew@dunslane.net> wrote: > FTR, I have put a lot of effort into maintaining and improving the > infrastructure over the years. And I don't think there is anything much > more important. So I'm going to put more effort in. And I'm not alone. > Andres, Alvaro, Noah and Thomas are some of those who have spent a lot > of effort on extending and improving our testing. I appreciate the work you've done, and the work others have done, and I'm sorry if my comments about the state of the project's infrastructure came across as a personal attack. Some of what is wrong here is completely outside of our control e.g. Perl is less popular. And even there, some people have done heroic work, like Noah stepping up to help maintain IPC::Run. And even with the stuff that is under our control, it's not that I don't like what you're doing. It's rather that I think we need more people doing it. For example, the fact that nobody's helping Thomas maintain this cfbot that we all have come to rely on, or helping him get that integrated into commitfest.postgresql.org, is a problem. You're not on the hook to do that, nor is anyone else. Likewise, the PostgreSQL::Test::whatever modules are mostly evolving when it's absolutely necessary to get some other patch committed, rather than anyone looking to improve them very much for their own sake. Maybe part of the problem, as Greg said, is that we don't do a good enough job advertising what the problems are or how people can help, but whatever the cause, it's not a very enjoyable experience, at least for me. But again, I don't blame you for any of that. You're clearly a big part of why it's going as well as it is! -- Robert Haas EDB: http://www.enterprisedb.com
On 2024-06-13 Th 17:23, Robert Haas wrote: > On Thu, Jun 13, 2024 at 4:54 PM Andrew Dunstan <andrew@dunslane.net> wrote: >> FTR, I have put a lot of effort into maintaining and improving the >> infrastructure over the years. And I don't think there is anything much >> more important. So I'm going to put more effort in. And I'm not alone. >> Andres, Alvaro, Noah and Thomas are some of those who have spent a lot >> of effort on extending and improving our testing. > I appreciate the work you've done, and the work others have done, and > I'm sorry if my comments about the state of the project's > infrastructure came across as a personal attack. Some of what is wrong > here is completely outside of our control e.g. Perl is less popular. > And even there, some people have done heroic work, like Noah stepping > up to help maintain IPC::Run. And even with the stuff that is under > our control, it's not that I don't like what you're doing. It's rather > that I think we need more people doing it. For example, the fact that > nobody's helping Thomas maintain this cfbot that we all have come to > rely on, or helping him get that integrated into > commitfest.postgresql.org, is a problem. You're not on the hook to do > that, nor is anyone else. Likewise, the PostgreSQL::Test::whatever > modules are mostly evolving when it's absolutely necessary to get some > other patch committed, rather than anyone looking to improve them very > much for their own sake. Maybe part of the problem, as Greg said, is > that we don't do a good enough job advertising what the problems are > or how people can help, but whatever the cause, it's not a very > enjoyable experience, at least for me. > > But again, I don't blame you for any of that. You're clearly a big > part of why it's going as well as it is! > Thank you, I'm not offended by anything you or anyone else has said. Clearly there are areas we can improve, and we need to be somewhat more proactive about it. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
On Thu, 13 Jun 2024 at 23:35, Andrew Dunstan <andrew@dunslane.net> wrote: > Clearly there are areas we can improve, and we need to be somewhat more > proactive about it. To follow that great suggestion, I updated meson wiki[1] after I realized some of the major gripes I had with the Perl tap test output were not actually caused by Perl but by meson: The main changes I made was using "-q --print-errorlogs" instead "-v", to reduce the enormous clutter in the output of the commands in the wiki to something much more reasonable. As well as adding examples on how to run specific tests [1]: https://wiki.postgresql.org/wiki/Meson#Test_related_commands
On Thu, Jun 13, 2024 at 1:04 PM Robert Haas <robertmhaas@gmail.com> wrote: > One caveat here, > perhaps, is that the focus of the work you've done up until now and > the things that I and other community members may want as a condition > of merging stuff may be somewhat distinct. You will have naturally > been focused on your goals rather than other people's goals, or so I > assume. Right. That's a risk I knew I was taking when I wrote it, so it's not going to offend me when I need to rewrite things. > I would be a bit wary of splitting it up over > too many different threads. It may well make sense to split it up, but > it will probably be easier to review if the core work to enable this > is one patch set on one thread where someone can read just that one > thread and understand the situation, rather than many threads where > you have to read them all. I'll try to avoid too many threads. But right now there is indeed just one thread (OAUTHBEARER) and it's way too much: - the introduction of pytest - a Construct-based manipulation of the wire protocol, including Wireshark-style network traces on failure - pytest fixtures which spin up libpq and the server in isolation from each other, relying on the Construct implementation to complete the seam - OAuth, which was one of the motivating use cases (but not the only one) for all of the previous items I really don't want to derail this thread with those. There are other people here with their own hopes and dreams (see: unconference notes), and I want to give them a platform too. > > That doesn't mean I want to > > do this without a plan; it just means that the plan can involve saying > > "this is not working and we can undo it" which makes the uncertainty > > easier to take. > > As a community, we're really bad at this. [...] I will carry the response to this to the next email. Thanks, --Jacob
On Thu, Jun 13, 2024 at 1:27 PM Robert Haas <robertmhaas@gmail.com> wrote: > > On Thu, Jun 13, 2024 at 4:06 PM Jacob Champion > <jacob.champion@enterprisedb.com> wrote: > > There was a four-step plan sketch at the end of that email, titled "A > > Plan". That was not intended to be "the final detailed plan", because > > I was soliciting feedback on the exact pieces people wanted to try to > > implement first, and I am not the God Emperor of Pytest. But it was > > definitely A Plan. > > Well, OK, now I feel a bit dumb. I guess I missed that or forgot about it. No worries. It's a really long thread. :D But also: do you have opinions on what to fill in as steps 2 (something we have no ability to test today) and 3 (something we do test today, but hate)? My vote for step 2 is "client and server separation", perhaps by testing libpq fallback against a server that claims support for different build-time options. I don't want to have a vote in step 3, because part of that step is proving that this framework can provide value for a part of the project I don't really know much about. > > I think this is way too much expectation for a v1 patch. If you were > > committing this by yourself, would you agree to develop the entirety > > of PostgreSQL::Test in a single commit, without the benefit of the > > buildfarm checking you as you went, and other people trying to write > > tests with it? > > Eh... I'm confused. PostgreSQL::Test::Cluster is more than half of the > code in that directory, and without it you wouldn't be able to write > most of the TAP tests that we have today. Well, in my defense, you said "PostgreSQL::Test::whatever", which I assumed meant all of it, including Kerberos.pm and SSL::Server and AdjustUpgrade and... That seemed like way too much to me (and still does!), but if that's not what you were arguing then never mind. Yes, Cluster.pm seems like a pretty natural thing to ask for. I imagine it's one of the first things we're going to need. And yet... > You would really want to > call this project done without having an equivalent? ...I have this really weird sneaking suspicion that, if a replacement of the end-to-end Perl acceptance tests can be made an explicit anti-goal in the short term, we might not necessarily need an "equivalent" for v1. I realize that seems bizarre, because of course we need a way to start the server if we want to test the server. But frankly, starting a server is Pretty Easy (tm), and Cluster.pm has to do a lot more than that because IMO it's designed for a variety of acceptance-oriented tasks. 3000+ lines! If there's widespread interest (as opposed to being just my own personal fever dream) in testing Postgres components as individual pieces rather than setting up the world, then I wonder if the functionality from Cluster.pm couldn't be pared down a lot. Maybe you don't need a centralized ->psql() or a ->command_ok() helper, because you're usually not trying to test psql and other utilities during your server-only tests. Maybe you can just stand up a standby without a primary and drive it via mock replication. Do you need quite as many "poll and wait for some asynchronous result" type things when you're not waiting for a result to cascade through a multinode system? Does something like (for example) ->pg_recvlogical_upto() really have to be implemented in our "core" fixtures or can it be done more easily by whoever needs that in the future? Maybe You Ain't Gonna Need It. If (he said, atop his naive idealistic soapbox) we can find a way to put off writing utilities until we write the tests that need them, without procrastinating, and without putting all of the negative externalities of that approach on the committers with low-quality copy-paste proliferation, and I'd like a pony while I'm at it, then I think the result might end up being pretty coherent and maintainable. Then not having "at least as much in-tree support for writing tests as we have today" for the very first commit would be a feature and not a bug. Now, maybe if the collective ability to do that existed, we would have done it already with Perl, but I do actually wonder whether that's true or not. Or, maybe, the very first suggestion for Step 3 will be something that needs absolutely everything in Cluster.pm. So be it; I can live without a pony. > You would really want to > call this project done without having an equivalent? (A cop-out but not-really-cop-out alternative answer to this question is that this project is not going to be "done" any more than Postgres will ever be "done", and that's part of what I'm arguing should be considered natural and okay. I understand that it is easier for me to take that stance when I am not on the hook for maintaining it, so I don't expect us to necessarily see eye-to-eye on it.) > > Can you elaborate on why that's not an okay outcome? > > Well, you just argued that it should be an okay outcome, and I do sort > of see your point, but I refer you to my earlier reply about the > difficulty of getting anything reverted in the culture as it stands. Earlier reply was: > As a community, we're really bad at this. Once something gets > committed, getting a consensus to revert it is really hard, especially > if a major release has happened meanwhile, but most of the time even > if it hasn't. It might be a little easier in this case, since after > all it's not a directly user-visible feature. But historically what > happens if somebody says "hey, there are six unfixed problems with > this feature!" is that everybody says "well, you're free to fix the > problems if you want, but you're not allowed to revert the feature." > And that is *exactly* how we end up with stuff like the current TAP > test framework: ripping that out would mean removing all the TAP tests > that depend on it, and that wouldn't have achieved consensus two > months after the feature went in, let alone today. Well... I don't know how to fix that. Here's a draft proposal after a few minutes of thought, which may need to be discarded after a few more minutes of thought. If there's agreement that New Tests -- not necessarily written in Python, but I selfishly hope they are -- exist on a probationary status, then maybe part of that is going to have to be an agreement: New features have to be able to have some minimum maintainability level *on the basis of the Perl tests only*, while the probationary period is in effect. It can't be the equivalent maintainability level, because that's either proof that the New Tests are giving us nothing, or proof that everyone is being forced to implement the exact same tests in both Perl and New Test. Neither is good. Since we're currently focused on end-to-end acceptance with Perl, that is probably a lower bar than what we'd maybe prefer, but I think that is the bar we have right now. It also exists as a forcing function to make sure that the additional tests are adding value over what we get with the Perl, which may paradoxically increase the chances of New Test success. (I can't tell if this is magical thinking or not.) So if a committer doesn't want responsibility for the feature if the new tests were deleted, they don't commit. Maybe that's unrealistic and too painful. It does increase the review requirements of committers quite a bit. It might disqualify my OAuth work (which is maybe evidence in its favor?). Maybe it increases the foot-in-the-door effect too much. Maybe there would have to be some trust-building where right now there is not? Not sure. > Now, it has been suggested to me by at least one other person involved > with the project that we need to be more open to the kind of thing > that you propose here: add experimental things and take them out if it > doesn't work out. I can definitely understand that this might be a > culturally better approach than what we currently do. So maybe that's > the way forward, but it is hard (at least for me) to get past the fear > of being the one left holding the bag, and I suspect that other > committers have similar fears. What exactly we should do about that, > I'm not sure. Yeah. > I have zero desire to write tests in Python. If I could convince > everyone here to spend their time and energy improving the stuff we > have in Perl instead of introducing a whole new test framework, I > would 100% do that. But I'm pretty sure that I can't, and I think the > project needs to pick from among realistic options rather than > theoretical ones. Said differently, it's not all about me. Then, for what it's worth: I really do want to make sure that your life, and the life of all the other committers, does not get significantly harder if this goes in. I don't think it will, but if I'm wrong, I want it to come back out, and then we can regroup or pivot entirely and move forward together. --Jacob
On Thu, Jun 13, 2024 at 8:12 PM Jacob Champion <jacob.champion@enterprisedb.com> wrote: > But also: do you have opinions on what to fill in as steps 2 > (something we have no ability to test today) and 3 (something we do > test today, but hate)? > > My vote for step 2 is "client and server separation", perhaps by > testing libpq fallback against a server that claims support for > different build-time options. I don't want to have a vote in step 3, > because part of that step is proving that this framework can provide > value for a part of the project I don't really know much about. I mean, both Perl and Python are Turing-complete. Anything you can do in one, you can do in the other, especially when you consider that we're not likely to accept too many dependencies on external Perl or Python modules. That's why I see this as nothing more or less than exercise in letting people use the programming language they prefer. We've talked about a libpq FFI interface, but it hasn't been done; now we're talking about maybe having a Python one. Perhaps we'll end up with both. Then you can imagine porting tests from one language to the other and the only difference is whether you'd rather have line noise before all of your variable names or semantically significant whitespace. I just don't believe in the idea that we're going to write one category of tests in one language and another category in another language. As soon as we open the door to Python tests, people are going to start writing the TAP tests that they would have written in Perl in Python instead. And if the test utilities that we have for Perl are not there for Python, then they'll either open code things for which they would have used a module, or they'll write a stripped-down version of the module that will then get built-up patch by patch until, 50 or 100 or 200 hours of committer-review later, it resembles the existing Perl module. And, if the committer pushes back and says, hey, why not write this test in Perl which already has all of this test infrastructure in place already, then the submitter will wander off muttering about how PostgreSQL committers are hidebound backward individuals who just try to ruin everybody's day. So as I see it, the only reasonable plan here if we want to introduce testing in Python (or C#, or Ruby, or Go, or JavaScript, or Lua, or LOLCODE) is to try to achieve a reasonable degree of parity between that language and Perl. Because then we can at least review the new infrastructure all at once, instead of incrementally spread across many patches written, reviewed, and committed by many different people. Now, I completely understand if you're not excited about getting sucked down that rabbit-hole, and maybe some other committer is going to see this differently than I do, and that's fine. But my view is that if you're not interested in doing all the work to let people do more or less the kind of stuff that they currently do in Perl in Python instead, then your alternative is to take the tests that you want to add and rewrite them in Perl. And I am fairly cetain that if you choose that option, it will save me, and a bunch of other committers, a whole lot of work, at least in the short term. If we add support for Python, we are going to end up having to do a lot of things twice for the next let's say ten to twenty years until somebody rewrites the remaining Perl tests in Python or whatever language is hot and cool by then. My opinion is that we need to be open to enduring that pain because we can't indefinitely hold our breath and insist on using obsolete tools for everything, but that doesn't mean that I don't think it will be painful. Consider the meson build system project. To get that committed, Andres had to make it do pretty much everything MSVC could do and mostly everything that configure could do, and the places where he didn't make it do everything configure could do remain sore spots that I, at least, am not happy about. And in that case, he also managed to get MSVC removed entirely, so that we do not have a larger number of build systems in total than we had before. Furthermore, the amount of code in buildsystem files (makefiles, meson.build) in a typical patch needing review is usually none or very little, whereas the amount of test code in a patch is sometimes quite large. I've come around to the belief that the meson work was worthwhile -- running tests is so much faster and nicer now -- but it was a ton of work to get done and impacted everyone in the development community, and I think the blast radius for this change is likely to be larger for the reasons suggested earlier in this paragraph. -- Robert Haas EDB: http://www.enterprisedb.com
On 2024-06-14 Fr 08:10, Robert Haas wrote:
We've talked about a libpq FFI interface, but it hasn't been done;
Hold my beer :-)
I just posted a POC for that.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
Robert Haas <robertmhaas@gmail.com> writes:
> I mean, both Perl and Python are Turing-complete. Anything you can do
> in one, you can do in the other, especially when you consider that
> we're not likely to accept too many dependencies on external Perl or
> Python modules. That's why I see this as nothing more or less than
> exercise in letting people use the programming language they prefer.
I think that's an oversimplified analysis. Sure, the languages are
both Turing-complete, but for our purposes here they are both simply
glue languages around some set of testing facilities. Some of those
facilities will be provided by the base languages (plus whatever
extension modules we choose to require) and some by code we write.
The overall experience of writing tests will be determined by the
testing facilities far more than by the language used for glue.
That being the case, I do agree with your point that Python
equivalents to most of PostgreSQL::Test will need to be built up PDQ.
Maybe they can be better than the originals, in features or ease of
use, but "not there at all" is not better.
But what I'd really like to see is some comparison of the
language-provided testing facilities that we're proposing
to depend on. Why is pytest (or whatever) better than Test::More?
I also wonder about integration of python-based testing with what
we already have. A significant part of what you called the meson
work had to do with persuading pg_regress, isolationtester, etc
to output test results in the common format established by TAP.
Am I right in guessing that pytest will have nothing to do with that?
Can we even manage to dump perl and python test scripts into the same
subdirectory and sort things out automatically? I'm definitely going
to be -1 for a python testing feature that cannot integrate with what
we have because it demands its own control and result-collection
infrastructure.
regards, tom lane
Hi,
On 2024-06-14 11:49:29 -0400, Tom Lane wrote:
> I also wonder about integration of python-based testing with what
> we already have. A significant part of what you called the meson
> work had to do with persuading pg_regress, isolationtester, etc
> to output test results in the common format established by TAP.
FWIW, meson's testrunner doesn't require TAP, the lowest common denominator is
just an exit code. However, for things that run many "sub" tests, it's a lot
nicer if the failures can be reported more granularly than just "the entire
testsuite failed".
Meson currently supports:
exitcode: the executable's exit code is used by the test harness to record the outcome of the test).
tap: Test Anything Protocol.
gtest (since 0.55.0): for Google Tests.
rust (since 0.56.0): for native rust tests
> Am I right in guessing that pytest will have nothing to do with that?
Looks like there's a plugin for pytest to support tap as output:
https://pypi.org/project/pytest-tap/
However, it's not available as a debian package. I know that some folks just
advocate installing dependencies via venv, but I personally don't think
that'll fly. For one, it'll basically prevent tests being run by packagers.
> Can we even manage to dump perl and python test scripts into the same
> subdirectory and sort things out automatically?
That shouldn't be much of a problem.
> I'm definitely going to be -1 for a python testing feature that cannot
> integrate with what we have because it demands its own control and
> result-collection infrastructure.
Dito.
Greetings,
Andres Freund
On 2024-06-14 09:11:11 -0700, Andres Freund wrote: > On 2024-06-14 11:49:29 -0400, Tom Lane wrote: > > Am I right in guessing that pytest will have nothing to do with that? > > Looks like there's a plugin for pytest to support tap as output: > https://pypi.org/project/pytest-tap/ > > However, it's not available as a debian package. I know that some folks just > advocate installing dependencies via venv, but I personally don't think > that'll fly. For one, it'll basically prevent tests being run by packagers. If this were the blocker, I think we could just ship an output adapter ourselves. pytest-tap is not a lot of code: https://github.com/python-tap/pytest-tap/blob/main/src/pytest_tap/plugin.py So either vendoring it or just writing an even simpler version ourselves seems entirely feasible.
On Thu, Jun 13, 2024 at 1:08 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
But Perl is at the next level of unmaintained infrastructure. It is
actually clear how you can contribute to it, but still no new
community members actually want to contribute to it. Also, it's not
only unmaintained by us but it's also pretty much unmaintained by the
upstream community.
I am not happy with the state of Perl, as it has made some MAJOR missteps along the way, particularly in the last 5 years. But can we dispel this strawman? There is a difference between "unpopular" and "unmaintained". The latest version of Perl was released May 20, 2024. The latest release of Test::More was April 25, 2024. Both are heavily used. Just not as heavily as they used to be. :)
Cheers,
Greg
On Fri, 14 Jun 2024 at 22:33, Greg Sabino Mullane <htamfids@gmail.com> wrote: > I am not happy with the state of Perl, as it has made some MAJOR missteps along the way, particularly in the last 5 years.But can we dispel this strawman? There is a difference between "unpopular" and "unmaintained". The latest version ofPerl was released May 20, 2024. The latest release of Test::More was April 25, 2024. Both are heavily used. Just not asheavily as they used to be. :) Sorry, yes I exaggerated here. Looking at the last Perl changelog[1] it's definitely getting more new features and improvements than I had thought. Test::More on the other hand, while indeed still maintained, it's definitely not getting significant new feature development or improvements[2]. Especially when comparing it to pytest[3]. [1]: https://perldoc.perl.org/perldelta [2]: https://github.com/Test-More/test-more/blob/master/Changes [3]: https://docs.pytest.org/en/stable/changelog.html
On Fri, Jun 14, 2024 at 9:24 AM Robert Haas <robertmhaas@gmail.com> wrote: > For example, the fact that > nobody's helping Thomas maintain this cfbot that we all have come to > rely on, or helping him get that integrated into > commitfest.postgresql.org, is a problem. I've been talking to Magnus and Jelte about cfbot and we're hoping to have some good news soon...
On Fri, 14 Jun 2024 at 17:49, Tom Lane <tgl@sss.pgh.pa.us> wrote: > But what I'd really like to see is some comparison of the > language-provided testing facilities that we're proposing > to depend on. Why is pytest (or whatever) better than Test::More? Some advantages of pytest over Test::More: 1. It's much easier to debug failing tests using the output that pytest gives. A good example of this is on pytest its homepage[1] (i.e. it shows the value given to the call to inc in the error) 2. No need to remember a specific comparison DSL (is/isnt/is_deeply/like/ok/cmp_ok/isa_ok), just put assert in front of a boolean expression and your output is great (if you want to add a message too for clarity you can use: assert a == b, "the world is ending") 3. Very easy to postgres log files on stderr/stdout when a test fails. This might be possible/easy with Perl too, but we currently don't do that. So right now for many failures you're forced to traverse the build/testrun/... directory tree to find the logs. 4. Pytest has autodiscovery of test files and functions, so we probably wouldn't have to specify all of the exact test files anymore in the meson.build files. Regarding 2, there are ~150 checks that are using a suboptimal way of testing for a comparison. Mostly a lot that could use "like(..., ...)" instead of "ok(... ~= ...)" ❯ grep '\bok(.*=' **.pl | wc -l 151 [1]: https://docs.pytest.org/en/8.2.x/#a-quick-example
On 2024-06-14 Fr 18:11, Jelte Fennema-Nio wrote: > On Fri, 14 Jun 2024 at 17:49, Tom Lane<tgl@sss.pgh.pa.us> wrote: >> But what I'd really like to see is some comparison of the >> language-provided testing facilities that we're proposing >> to depend on. Why is pytest (or whatever) better than Test::More? > Some advantages of pytest over Test::More: > > 1. It's much easier to debug failing tests using the output that > pytest gives. A good example of this is on pytest its homepage[1] > (i.e. it shows the value given to the call to inc in the error) > 2. No need to remember a specific comparison DSL > (is/isnt/is_deeply/like/ok/cmp_ok/isa_ok), just put assert in front of > a boolean expression and your output is great (if you want to add a > message too for clarity you can use: assert a == b, "the world is > ending") > 3. Very easy to postgres log files on stderr/stdout when a test fails. > This might be possible/easy with Perl too, but we currently don't do > that. So right now for many failures you're forced to traverse the > build/testrun/... directory tree to find the logs. I see the fact that we stash the output in a file as a feature. Without it, capturing failure information in the buildfarm client would be more difficult, especially if there are multiple failures. So this is actually something I think we would need for any alternative framework. Maybe we need an environment setting that would output the regress_log_00whatever file to stderr on failure. That should be pretty easy to arrange in the END handler for PostgreSQL::Test::Utils. > 4. Pytest has autodiscovery of test files and functions, so we > probably wouldn't have to specify all of the exact test files anymore > in the meson.build files. I find this comment a bit ironic. We don't need to do that with the Makefiles, and the requirement to do so was promoted as a meson feature rather than a limitation, ISTR. > Regarding 2, there are ~150 checks that are using a suboptimal way of > testing for a comparison. Mostly a lot that could use "like(..., ...)" > instead of "ok(... ~= ...)" > ❯ grep '\bok(.*=' **.pl | wc -l > 151 Well, let's fix those. I would be tempted to use cmp_ok() for just about all of them. But the fact that Test::More has a handful of test primitives rather than just one strikes me as a relatively minor complaint. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
On Sat, 15 Jun 2024 at 16:45, Andrew Dunstan <andrew@dunslane.net> wrote: > I see the fact that we stash the output in a file as a feature. Without > it, capturing failure information in the buildfarm client would be more > difficult, especially if there are multiple failures. So this is > actually something I think we would need for any alternative framework. I indeed heard that the current behaviour was somehow useful to the buildfarm client. > Maybe we need an environment setting that would output the > regress_log_00whatever file to stderr on failure. That should be pretty > easy to arrange in the END handler for PostgreSQL::Test::Utils. That sounds awesome! But I'm wondering: do we really need a setting to enable/disable that? Can't we always output it to stderr on failure? If we output the log both to stderr and as a file, would that be fine for the build farm? If not, a setting should work. (but I'd prefer the default for that setting to be on in that case, it seems much easier to turn it off in the buildfarm client, instead of asking every developer to turn the feature on) > > 4. Pytest has autodiscovery of test files and functions, so we > > probably wouldn't have to specify all of the exact test files anymore > > in the meson.build files. > > > I find this comment a bit ironic. We don't need to do that with the > Makefiles, and the requirement to do so was promoted as a meson feature > rather than a limitation, ISTR. Now, I'm very curious why that would be considered a feature. I certainly have had many cases where I forgot to add the test file to the meson.build file. > > Regarding 2, there are ~150 checks that are using a suboptimal way of > > testing for a comparison. Mostly a lot that could use "like(..., ...)" > > instead of "ok(... ~= ...)" > > ❯ grep '\bok(.*=' **.pl | wc -l > > 151 > > > Well, let's fix those. I would be tempted to use cmp_ok() for just about > all of them. Sounds great to me. > But the fact that Test::More has a handful of test primitives rather > than just one strikes me as a relatively minor complaint. It is indeed a minor paper cut, but paper-cuts add up. Honestly, my primary *objective* complaint about our current test suite, is that when a test fails, it's very often impossible for me to understand why the test failed, by only looking at the output of "meson test". I think logging the postgres log to stderr for Perl, as you proposed, would significantly improve that situation. I think the only thing that we cannot get from Perl Test::More that we can from pytest, is the fancy recursive introspection of the expression that pytest shows on error. Apart from that my major *subjective* complaint is that I very much dislike writing Perl code. I'm slow at writing it and I don't (want to) improve at it because I don't have reasons to use it except for Postgres tests. So currently I'm not really incentivised to write more tests than the bare minimum, help improve the current test tooling, or add new testing frameworks for things we currently cannot test. Afaict, there's a significant part of our current community who feel the same way (and I'm pretty sure every sub-30 year old person who newly joins the community would feel the exact same way too). As a project I think we would like to have more tests, and to have more custom tooling to test things that we currently cannot (e.g. oauth or manually messing with the wire-protocol). I think the only way to achieve that is by encouraging more people to work on these things. I very much appreciate that you and others are improving our Perl tooling, because that makes our current tests easier to work with. But I don't think it significantly increases the willingness to write tests or test-tooling for people that don't want to write Perl in the first place. So I think the only way to get more people involved in contributing tests and test-tooling is by allowing testing in another language than Perl (but also still allow writing tests in Perl). Even if that means that we have two partially-overlapping test frameworks, that are both easy to use for different things. In my view that's even a positive thing, because that means we are able to test more with two languages than we would be able to with either one (and it's thus useful to have both). And I agree with Robbert that Python seems like the best choice for this other language, given its current popularity level. But as I said before, I'm open to other languages as well.
On Sat, Jun 15, 2024 at 12:48 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote: > Honestly, my primary *objective* complaint about our current test > suite, is that when a test fails, it's very often impossible for me to > understand why the test failed, by only looking at the output of > "meson test". I think logging the postgres log to stderr for Perl, as > you proposed, would significantly improve that situation. I think the > only thing that we cannot get from Perl Test::More that we can from > pytest, is the fancy recursive introspection of the expression that > pytest shows on error. This surprises me. I agree that the current state of affairs is kind of annoying, but the contents of regress_log_whatever are usually quite long. Printing all of that out to standard output seems like it's just going to flood the terminal with output. I don't think I'd be a fan of that change. I think I basically agree with all the nearby comments about how the advantages you cite for Python aren't, I don't know, entirely compelling. Switching from ok() to is() or cmp_ok() or like() is minor stuff. Where the output goes is minor stuff. The former can be fixed, and the latter can be worked around with scripts and aliases. The one thing I know about that *I* think is a pretty big problem about Perl is that IPC::Run is not really maintained. But I wonder if the solution to that is to do something ourselves instead of depending on IPC::Run. Beyond that, I think this is just a language popularity contest. -- Robert Haas EDB: http://www.enterprisedb.com
Hi, On 2024-06-15 10:45:16 -0400, Andrew Dunstan wrote: > > 4. Pytest has autodiscovery of test files and functions, so we > > probably wouldn't have to specify all of the exact test files anymore > > in the meson.build files. > > > I find this comment a bit ironic. We don't need to do that with the > Makefiles, and the requirement to do so was promoted as a meson feature > rather than a limitation, ISTR. The reason its good to have the list of tests somewhere explicit is that we have different types of test. With make, there is a single target for all tap tests. If you want to run tests concurrently, make can only schedule the tap tests at the granularity of a directory. If you want concurrency below that, you need to use concurrency on the prove level. But that means that you have extremely varying concurrency, depending on whether make runs targets that have no internal concurrency or make runs e.g. the recovery tap tests. I don't think we should rely on global test discovery via pytest. That'll lead to uncontrollable concurrency again, which means much longer test times. We'll always have different types of tests, just scheduling running them via "top-level" tools for different test types just won't work well. That's not true for many projects where tests have vastly lower resource usage. Greetings, Andres Freund
On Sat, 15 Jun 2024 at 19:27, Robert Haas <robertmhaas@gmail.com> wrote: > This surprises me. I agree that the current state of affairs is kind > of annoying, but the contents of regress_log_whatever are usually > quite long. Printing all of that out to standard output seems like > it's just going to flood the terminal with output. I don't think I'd > be a fan of that change. I think at the very least the locations of the different logs should be listed in the output.
On Sat, Jun 15, 2024 at 12:48 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
Afaict, there's a significant part of our current community who feel the same way (and I'm pretty sure every sub-30 year old person who
newly joins the community would feel the exact same way too).
Those young-uns are also the same group who hold their nose when coding in C, and are always clamoring for rewriting Postgres in Rust. And before that, C++. And next year, some other popular language that is clearly better and more popular than C.
And I agree with Robbert that Python seems like the best choice for this other language, given its current popularity level. But as I said
before, I'm open to other languages as well.
Despite my previous posts, I am open to other languages too, including Python, but the onus is really on the new language promoters to prove that the very large amount of time and trouble is worth it, and worth it for language X.
Cheers,
Greg
On Fri, Jun 14, 2024 at 5:09 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
Test::More on the other hand, while indeed still maintained, it's
definitely not getting significant new feature development or
improvements[2]. Especially when comparing it to pytest[3].
That's fair, although it's a little hard to tell if the lack of new features is because they are not needed for a stable, mature project, or because few people are asking for and developing new features. Probably a bit of both. But I'll be the first to admit Perl is dying; I just don't know what should replace it (or how - or when). Python has its quirks, but all languages do, and your claim that it will encourage more and easier test writing by developers is a good one.
Cheers,
Greg
On Sat, Jun 15, 2024 at 5:53 PM Greg Sabino Mullane <htamfids@gmail.com> wrote: > > On Sat, Jun 15, 2024 at 12:48 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote: >> >> Afaict, there's a significant part of our current community who feel the same way (and I'm pretty sure every sub-30 yearold person who >> newly joins the community would feel the exact same way too). > > > Those young-uns are also the same group who hold their nose when coding in C, and are always clamoring for rewriting Postgresin Rust. And before that, C++. And next year, some other popular language that is clearly better and more popularthan C. Writing a new test framework in a popular language that makes it more likely that more people will write more tests and test infrastructure is such a completely different thing than suggesting we rewrite Postgres in Rust that I feel that this comparison is unfair and, frankly, a distraction from the discussion at hand. - Melanie
On Sat, Jun 15, 2024 at 6:00 PM Melanie Plageman <melanieplageman@gmail.com> wrote: > > Those young-uns are also the same group who hold their nose when coding in C, and are always clamoring for rewritingPostgres in Rust. And before that, C++. And next year, some other popular language that is clearly better and morepopular than C. > > Writing a new test framework in a popular language that makes it more > likely that more people will write more tests and test infrastructure > is such a completely different thing than suggesting we rewrite > Postgres in Rust that I feel that this comparison is unfair and, > frankly, a distraction from the discussion at hand. I don't really agree with this. We have been told before that we would attract more developers to our community if only we allowed backend code to be written in C++ or Rust, and that is not altogether a different thing than saying that we would attract more test developers if only we allowed test code to be written in Python or whatever. The difference is one of degree rather than of kind. We have a lot more backend code than we do test code, I'm fairly sure, and our tests are more self-contained: it's not *as* problematic if some tests are written in one language and others in another as it would be if different parts of the backend used different languages, and it wouldn't be *as* hard if at some point we decided we wanted to convert all remaining code to the new language. So, I have a much harder time imagining that we would start allowing a new language for backend code than that we would start allowing a new language for tests, but I don't think the issues are fundamentally different. But that said, I'm not sure the programming language is the real issue. If I really wanted to participate in an open source project, I'd probably be willing to learn a new programming language to do that. Maybe some people wouldn't, but I had to learn a whole bunch of them in college, and learning one more doesn't sound like the biggest of deals. But, would I feel respected and valued as a participant in that project? Would I have to use weird tools and follow arcane and frustrating processes? If I did, *that* would make me give up. I don't want to say that the choice of programming language doesn't matter at all, but it seems to me that it might matter more because it's a symptom of being unwilling to modernize things rather than for its own sake. -- Robert Haas EDB: http://www.enterprisedb.com
Hi Greg, Jelte,
On Sat, 15 Jun 2024 at 23:53, Greg Sabino Mullane <htamfids@gmail.com> wrote:
>
> On Sat, Jun 15, 2024 at 12:48 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
>>
>> Afaict, there's a significant part of our current community who feel the same way (and I'm pretty sure every sub-30 year old person who
>> newly joins the community would feel the exact same way too).
I feel I'm still relatively new (started in the past 4 years) and I have quite some time left before I hit 30 years of age.
>
> On Sat, Jun 15, 2024 at 12:48 PM Jelte Fennema-Nio <postgres@jeltef.nl> wrote:
>>
>> Afaict, there's a significant part of our current community who feel the same way (and I'm pretty sure every sub-30 year old person who
>> newly joins the community would feel the exact same way too).
I feel I'm still relatively new (started in the past 4 years) and I have quite some time left before I hit 30 years of age.
I don't specifically feel the way you describe, nor do I think I've ever really felt like that in the previous nearly 4 years of hacking. Then again, I'm not interested in testing frameworks, so I don't feel much at all about which test frameworks we use.
> Those young-uns are also the same group who hold their nose when coding in C, and are always clamoring for rewriting Postgres in Rust.
> Those young-uns are also the same group who hold their nose when coding in C, and are always clamoring for rewriting Postgres in Rust.
Could you point me to one occasion I have 'always' clamored for this, or any of "those young-uns" in the community? I may not be a huge fan of C, but rewriting PostgreSQL in [other language] is not on the list of things I'm clamoring for. I may have given off-hand mentions that [other language] would've helped in certain cases, sure, but I'd hardly call that clamoring.
Kind regards,
Matthias van de Meent
Hi Jacob, > For the v18 cycle, I would like to try to get pytest [1] in as a > supported test driver, in addition to the current offerings. > > (I'm tempted to end the email there.) Huge +1 from me and many thanks for working on this. Two cents from me. I spent a fair part of my career writing in Perl. Personally I like the language and often find it more convenient for the tasks I'm working on than Python. This being said, there were several projects I was involved in where we had to choose a scripting language. In all the cases there was a strong push-back against Perl and Python always seemed to be a common denominator for everyone. So I ended up with the rule of thumb to use Perl for projects I'm working on alone and Python otherwise. Although the objective reality in the entire industry is unknown to me I spoke to many people whose observations were similar. We could speculate about the reasons why people seem to prefer Python (good IDE support*, unique** libraries like Matplotlib / NumPy / SciPy, ...) but honestly I don't think they are extremely relevant in this discussion. I believe supporting Python in our test infrastructure will attract more contributors and thus would be a good step for the project in the long run. * including PyTest integration ** citation needed -- Best regards, Aleksander Alekseev
On Sun, 16 Jun 2024 at 23:04, Robert Haas <robertmhaas@gmail.com> wrote: > > On Sat, Jun 15, 2024 at 6:00 PM Melanie Plageman > <melanieplageman@gmail.com> wrote: > > Writing a new test framework in a popular language that makes it more > > likely that more people will write more tests and test infrastructure > > is such a completely different thing than suggesting we rewrite > > Postgres in Rust that I feel that this comparison is unfair and, > > frankly, a distraction from the discussion at hand. > > I don't really agree with this. > <snip> > it's not *as* problematic if some tests are > written in one language and others in another as it would be if > different parts of the backend used different languages, and it > wouldn't be *as* hard if at some point we decided we wanted to convert > all remaining code to the new language. Honestly, it sounds like you actually do agree with each other. It seems you interpreted Melanie her use of "thing" as "people wanting to use Rust/Python in the Postgres codebase" while I believe she probably meant "all the problems and effort involved in the task making that possible''. And afaict from your response, you definitely agree that making it possible to use Rust in our main codebase is a lot more difficult than for Python for our testing code. > But, would I feel respected and valued as a participant in > that project? Would I have to use weird tools and follow arcane and > frustrating processes? If I did, *that* would make me give up. I don't > want to say that the choice of programming language doesn't matter at > all, but it seems to me that it might matter more because it's a > symptom of being unwilling to modernize things rather than for its own > sake. I can personally definitely relate to this (although I wouldn't frame it as strongly as you did). Postgres development definitely requires weird tools and arcane processes (imho) when compared to most other open source projects. The elephant in the room is of course the mailing list development flow. But we have some good reasons for using that[^1]. But most people have some limit on the amount of weirdness they are willing to accept when wanting to contribute, and the mailing list pushes us quite close to that limit for a bunch of people already. Any additional weird tools/arcane processes might push some people over that limit. We've definitely made big improvements in modernizing our development workflow over the last few years though: We now have CI (cfbot), a modern build system (meson), and working autoformatting (requiring pgindent on commit). These improvements have been very noticeable to me, and I think we should continue such efforts. I think allowing people to write tests in Python is one of the easier improvements that we can make. [^1]: Although I think those reasons apply much less to the documentation, maybe we could allow github contributions for just those.
On 2024-06-17 Mo 4:27 AM, Matthias van de Meent wrote: > Hi Greg, Jelte, > > On Sat, 15 Jun 2024 at 23:53, Greg Sabino Mullane <htamfids@gmail.com> > wrote: > > > Those young-uns are also the same group who hold their nose when > coding in C, and are always clamoring for rewriting Postgres in Rust. > > Could you point me to one occasion I have 'always' clamored for this, > or any of "those young-uns" in the community? I may not be a huge fan > of C, but rewriting PostgreSQL in [other language] is not on the list > of things I'm clamoring for. I may have given off-hand mentions that > [other language] would've helped in certain cases, sure, but I'd > hardly call that clamoring. > > Greg was being a but jocular here. I didn't take him seriously. But there's maybe a better case to make the point he was making. Back in the dark ages we used a source code control system called CVS. It's quite unlike git and has a great many limitations and uglinesses, and there was some pressure for us to move off it. If we had done so when it was first suggested, we would probably have moved to using Subversion, which is rather like CVS with many of the warts knocked off. Before long, some distributed systems like Mercurial and git came along, and we, like most of the world, chose git. Thus by waiting and not immediately doing what was suggested we got a better solution. Moving twice would have been ... painful. I have written Python in the past. Not a huge amount, but it doesn't feel like a foreign country to me, just the next town over instead of my immediate neighbourhood. We even have a python script in the buildfarm server code (not written by me). I'm sure if we started writing tests in Python I would adjust. But I think we need to know what the advantages are, beyond simple language preference. And to get to an equivalent place for Python that we are at with perl will involve some work. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
(slowly catching up from the weekend email backlog) On Fri, Jun 14, 2024 at 5:10 AM Robert Haas <robertmhaas@gmail.com> wrote: > I mean, both Perl and Python are Turing-complete. Tom responded to this better than I could have, but I don't think this is a helpful statement. In fact I opened the unconference session with it and immediately waved it away as not-the-point. > I just don't believe in the idea that we're going to write one > category of tests in one language and another category in another > language. You and I will probably not agree, then, because IMO we already do that. SQL behavior is tested in SQL via pg_regress characterization tests. End-to-end tests are written in Perl. Lower-level tests are often written in C (and, unfortunately, then driven in Perl instead of C; see above mail to Noah). I'm fundamentally a polyglot tester by philosophy, so I don't see careful use of multiple languages as an inherent problem to be solved. They increase complexity (considerably so!) but generally provide sufficient value to offset the cost. > As soon as we open the door to Python tests, people are > going to start writing the TAP tests that they would have written in > Perl in Python instead. There's a wide spectrum of opinions between yours (which I will cheekily paraphrase as "people will love testing in Python so much they'll be willing to reinvent all of the wheels" -- which in the short term would increase maintenance cost but in the long term sounds like a very good problem to have), and people who seem to think that new test suite infrastructure would sit unused because no one wants to write tests anyway (to pull a strawman out of some hallway conversations at PGConf.dev). I think the truth is probably somewhere in the middle? My prior mail was an attempt to bridge the gap between today and the medium term, by introducing a series of compromises and incremental steps in response to specific fears. We can jump forward to the end state and try to talk about it, but I don't control the end state and I don't have a crystal ball. > So as I see > it, the only reasonable plan here if we want to introduce testing in > Python (or C#, or Ruby, or Go, or JavaScript, or Lua, or LOLCODE) is > to try to achieve a reasonable degree of parity between that language > and Perl. Because then we can at least review the new infrastructure > all at once, instead of incrementally spread across many patches > written, reviewed, and committed by many different people. I don't at all believe that a test API which is ported en masse as a horizontal layer, without motivating vertical slices of test functionality, is going to be fit for purpose. And "written, reviewed, and committed by many different people" is a feature for me, not a bug. One of the goals of the thread is to encourage more community test writing than we currently have. Otherwise, I could have kept silent (I am very happy with my personal suite and have been comfortably maintaining it for a while). I am trying to build community momentum around a pain point that is currently rusted in place. > Consider the meson build system project. To get that committed, Andres > had to make it do pretty much everything MSVC could do and mostly > everything that configure could do I think some lessons can be pulled from that, but fundamentally that's a port of the build infrastructure done by a person with a commit bit. There are some pretty considerable differences. (And even then, it wasn't "done" with the first volley of patches, right?) Thanks, --Jacob
On Fri, Jun 14, 2024 at 8:49 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I think that's an oversimplified analysis. Sure, the languages are
> both Turing-complete, but for our purposes here they are both simply
> glue languages around some set of testing facilities. Some of those
> facilities will be provided by the base languages (plus whatever
> extension modules we choose to require) and some by code we write.
> The overall experience of writing tests will be determined by the
> testing facilities far more than by the language used for glue.
+1. As an example, the more extensive (and high-quality) a language's
standard library, the more tests you'll be able to write. Convincing
committers to adopt a new third-party dependency is hard (for good
reason); the strength of the standard library should be considered as
a point of technical comparison.
> That being the case, I do agree with your point that Python
> equivalents to most of PostgreSQL::Test will need to be built up PDQ.
> Maybe they can be better than the originals, in features or ease of
> use, but "not there at all" is not better.
There's a wide gulf between "not there at all" and "reimplement it all
as a prerequisite for v1" as Robert proposed. I'm arguing for a middle
ground.
> But what I'd really like to see is some comparison of the
> language-provided testing facilities that we're proposing
> to depend on. Why is pytest (or whatever) better than Test::More?
People are focusing a lot on failure reporting, and I agree with them,
but I did try to include more than just that in my OP.
I'll requote what I personally think is the #1 killer feature of
pytest, which prove and Test::More appear to be unable to accomplish
on their own: configurable isolation of tests from each other via
fixtures [1].
Problem 1 (rerun failing tests): One architectural roadblock to this
in our Test::More suite is that tests depend on setup that's done by
previous tests. pytest allows you to declare each test's setup
requirements via pytest fixtures, letting the test runner build up the
world exactly as it needs to be for a single isolated test. These
fixtures may be given a "scope" so that multiple tests may share the
same setup for performance or other reasons.
When I'm doing red-to-green feature development (e.g. OAuth) or
green-to-green refactoring (e.g. replacement of libiddawc with libcurl
in OAuth), quick cycle time and reduction of noise is extremely
important. I want to be able to rerun just the single red test I care
about before moving on.
(Tests may additionally be organized with custom attributes. My OAuth
suite contains tests that must run slowly due to mandatory timeouts;
I've marked them as slow, and they can be easily skipped from the test
runner.)
2. The ability to break into a test case with the built-in debugger
[2] is also fantastic for quick red-green work. Much better than
print() statements.
(Along similar lines, even the ability to use Python's built-in REPL
increases velocity. Python users understand that they can execute
`python3` and be dropped into a sandbox to try out syntax or some
unfamiliar library. Searching for how to do this in Perl results in a
handful of custom-built scripts; people here may know which to use as
a Perl monk, sure, but the point is to make it easy for newcomers to
write tests.)
> I also wonder about integration of python-based testing with what
> we already have. A significant part of what you called the meson
> work had to do with persuading pg_regress, isolationtester, etc
> to output test results in the common format established by TAP.
> Am I right in guessing that pytest will have nothing to do with that?
Andres covered this pretty well. I will note that I had problems with
pytest-tap itself [3], and I'm unclear whether that represents a bug
in pytest-tap or a bug in pytest.
Thanks,
--Jacob
[1] https://docs.pytest.org/en/stable/how-to/fixtures.html
[2] https://docs.pytest.org/en/stable/how-to/failures.html
[3] https://github.com/python-tap/pytest-tap/issues/30