Hi,
I have two buildfarm animals running tests with -DCLOBBER_CACHE_ALWAYS,
and both of them started having issues with timeouts
Last file mtime in snapshot: Sat Nov 29 21:41:49 2025 GMT
===================================================
timed out after 14400 secs
The animals are avocet and trilobite, and it's actually a single VM,
with one animal for gcc or clang.
It's fine that these tests take quite a bit of time, but I'm puzzled why
it started to fail. And it seems the stats_ext regression test got much
slower at the end of September.
The last couple runs from avocet have these durations:
11:44:42 ago timedout [713d9a8] (01:54:23) Details
7 days 23:33:36 ago OK [bc32a12] (34:19:40) Config
16 days 12:19:09 ago OK [* 23792] (34:32:51) Config
63 days 03:55:34 ago OK [91df046] (31:14:46) Config
71 days 10:57:06 ago OK [9fc7f6a] (31:51:02) Config
79 days 05:46:56 ago OK [454c046] (19:35:25) Config
87 days 00:36:55 ago OK [06473f5] (19:31:16) Config
94 days 19:50:32 ago OK [f727b63] (19:05:05) Config
102 days 14:43:13 ago OK [ef03ea0] (19:09:05) Config
...
So it was ~19h for a while (started at ~14h about 4y back). And then
between September 14 and 22 it jumped to ~32h. Which seems like a lot.
And that seems to be due to stats_ext changing from
ok 157 + stats_ext 855865 ms
to
ok 157 + stats_ext 15158426 ms
That's ~14300 seconds, i.e. ~4h. Pretty close to the timeout limit. The
machine is running the test with two locales, so there are multiple runs
of this regression test.
Any ideas what might be causing this? I see exactly the same issue on
the other machine (trilobite). There were no upgrades or other changes
around that time, and no other regression test has this issue.
regards
--
Tomas Vondra
