On Mon, Jul 22, 2024 at 11:48 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I'm a little suspicious
> of using it for tests that merely take an unreasonable amount of
> time --- to me, that indicates laziness on the part of the test
> author.
Laziness would have been not bothering to develop a TAP test for this
at all. Going to the trouble of creating one and not being able to
make it as fast or as stable as everybody would like is just being
human.
I never quite know what to do about TAP testing for issues like this.
Ideally, we want a test case that runs quickly, is highly stable, is
perfectly sensitive to the bug being fixed, and has a reasonable
likelihood of being sensitive to future bugs of the same ilk. But such
a test case need not exist, and even if it does, it need not be the
case that any of us are able to find it. Or maybe finding it is
possible but will take an unreasonable amount of time: if it took a
committer six months to come up with such a test case for this bug,
would that be worth it, or just overkill? I'd say overkill: I'd rather
have that committer working on other stuff than spending six months
trying to craft the perfect test case for a bug that's already fixed.
Also, this particular bug seems to require a very specific combination
of circumstances in order to trigger it. So the test gets complicated.
As mentioned, that makes it harder to get the test case fast and
stable, but it also reduces the chances that the test case will ever
find anything. I don't think that this will be the last time we make a
mistake around VACUUM's xmin handling, but the next mistake may well
require an equally baroque but *different* setup to cause a problem. I
hate to come to the conclusion that we just shouldn't test for this,
but I don't think it's fair to send Melanie off on a wild goose chase
looking for a perfect test case that may not realistically exist,
either.
--
Robert Haas
EDB: http://www.enterprisedb.com
