Re: Improving tracking/processing of buildfarm test failures - Mailing list pgsql-hackers

02.07.2024 15:00, Alexander Lakhin wrote:
>
> One month later,  I'd like to summarize failures that I've investigated
> and classified during June, 2024 on the aforementioned wiki page.
> (Maybe it would make sense to issue a monthly report with such information
> in the future.)

Please take a look at July report on the buildfarm failures:
# SELECT br, count(*) FROM failures WHERE dt >= '2024-07-01' AND
  dt < '2024-08-01' GROUP BY br;

REL_12_STABLE: 11
REL_13_STABLE: 9
REL_14_STABLE: 7
REL_15_STABLE: 10
REL_16_STABLE: 9
REL_17_STABLE: 68
HEAD: 106
-- Total: 220
(Counting test failures only, excluding indent-check, Configure, Build
errors.)

# SELECT COUNT(*) FROM (SELECT DISTINCT issue_link FROM failures WHERE
  dt >= '2024-07-01' AND dt < '2024-08-01');
40

# SELECT issue_link, count(*) FROM failures WHERE dt >= '2024-07-01' AND
  dt < '2024-08-01' GROUP BY issue_link ORDER BY 2 DESC LIMIT 9;

https://www.postgresql.org/message-id/20240404170055.qynecay7szu3dgvu@awork3.anarazel.de: 29
-- An environmental issue

https://www.postgresql.org/message-id/a9a97e83-9ec8-5de5-bf69-80e9560f5345@gmail.com: 20
-- Probably fixed

https://www.postgresql.org/message-id/1545399.1720554797@sss.pgh.pa.us: 11
-- Fixed

https://www.postgresql.org/message-id/4db099c8-4a52-3cc4-e970-14539a319466@gmail.com: 9

https://www.postgresql.org/message-id/db093cce-7eec-8516-ef0f-891895178c46@gmail.com: 8
-- An environmental issue; probably fixed

https://www.postgresql.org/message-id/b2037a8d-fe6b-d299-da17-ff5f3214e648@gmail.com: 8

https://www.postgresql.org/message-id/3e2cbd24-f45e-4b2b-ba83-8149214f0a4d@dunslane.net: 8
-- Fixed

https://www.postgresql.org/message-id/68de6498-0449-a113-dd03-e198dded0bac@gmail.com: 8
-- Fixed

https://www.postgresql.org/message-id/3618203.1722473994@sss.pgh.pa.us: 8
-- Fixed

# SELECT count(*) FROM failures WHERE dt >= '2024-07-01' AND
  dt < '2024-08-01' AND issue_link IS NULL; -- Unsorted/unhelpful failures
17

And one more metric, that might be useful, but it requires also time
analysis — short-lived (eliminated immediately) failures: 83

I also wrote a simple script (see attached) to check for unknown buildfarm
failures using "HTML API", to make sure no failures missed. Surely, it
could be improved in many ways, but I find it rather useful as-is.

Best regards,
Alexander