Re: RFC: adding pytest as a supported test framework - Mailing list pgsql-hackers

On Thu, Jun 13, 2024 at 11:11 AM Robert Haas <robertmhaas@gmail.com> wrote:
> I feel like I already agreed to this in a previous email and you're
> continuing to argue with me as if I were disagreeing.

I also think that maybe arguments are starting to sail past each
other, and the temperature is starting to climb. (And Jelte may be
arguing to all readers of the thread, rather than just a single
individual. It's hard to tell with the list format.) And now I see
that there's another email that came in while I was writing this, but
I think I'm going to have to send this as-is because I can't write
emails that fast.

> I also agree with this. I'm just not super optimistic about how much
> of that will actually happen. And I'd like to hear you acknowledge
> that concern and think about whether it can be addressed in some way,
> instead of just repeating that we should do it anyway. Because I agree
> we probably should do it anyway, but that doesn't mean I wouldn't like
> to see the downsides mitigated as much as we can.

Okay, +1.

> In particular, if
> the proposal is exactly "let's add the smallest possible patch that
> enables people to write tests in Python and then add a few new tests
> in Python while leaving almost everything else in Perl, with no
> migration plan and no clear vision of how the Python support ever gets
> any better than the minimum stub that is proposed for initial commit,"
> then I don't know that I can vote for that plan.

(that's not the proposal and I know/think you know that but having my
original email twisted into that is making me feel a bit crispy)

I do not want to migrate, and I have stated so multiple times, which
is why I have not proposed a migration plan. Other committers have
already expressed resistance to the idea that we would rewrite the
Perl stuff. I think they're right. I think we should not. I think we
should accept the cost of both Perl and something else for the
near-to-medium future, as long as the "something else" gives us value
offsetting the additional cost.

> Honestly, that sounds
> like very little work for the person proposing that minimal patch and
> a whole lot of work for the rest of the community later on, and the
> evidence is not in favor of volunteers showing up to take care of that
> work.

Okay, cool. Here: as the person who is 100% signing himself up to do
that, time for me to jump in.

I have an entire 6000-something-line suite of protocol tests that has
been linked like four times above. It does something fundamentally
different from the end-to-end Perl suite; it is not a port. It is far
from perfect and I do not want to pressure people to adopt it as-is,
which is why I have not. In this thread, I am offering it solely as
evidence that I have follow-up intent.

But I might get hit by a bus. Or, as far as anyone except me knows, I
might lose interest after things get hard, which would be sad. Which
is why my very first proposal was to add an entry point that can be
reverted. The suite is not going to infect the codebase any more than
the Perl does. A revert will involve pulling the Meson test entry
code, and deleting all pytest subdirectories (of which there is only
one, IIRC, in my OAuth suite).

> The plan should be more front-loaded than that: enough initial
> development should get done by the people making the proposal that if
> the work stops after, we don't have another big mess on our hands.

For me personally, the problem is the opposite. I have done _so much_
initial development by myself that there's no way it could ever be
reviewed and accepted. But I had to do that to get meaningful
development done in my style of work, which is focused on security and
testability and verifiable implementation.

I am trying to carve off pieces of that and say "hey, does this look
nice to anyone else?" That will take time, probably over multiple
different threads. In the meantime, I don't want to be a serialization
point for other people who are excited about trying new testing
methods, because very few people are currently doing the exact kind of
work I am doing. They may want to do other things, as evidenced by the
thread contents. At least one committer would have to sign up to be a
serialization point, unfortunately, but I think that's going to be
true regardless of plan, if we want multiple non-committer members of
the community to be involved instead of just one torch-bearer.

Because of how many moving parts and competing interests and personal
disagreements there are, I am firmly in the camp of "try something
that many people think *might* work better, that can be undone if it
sucks, and iterate on it." I want to build community momentum, because
I think that's a pretty effective way to change the cultural norms
that you're saying you're frustrated with. That doesn't mean I want to
do this without a plan; it just means that the plan can involve saying
"this is not working and we can undo it" which makes the uncertainty
easier to take.

--Jacob