Low frequency AIO checksum corruption on buildfarm member fruitcrow / GNU Hurd - Mailing list pgsql-hackers

Hi,

Every few days/weeks, a run on my Debian GNU/HUrd buildfarm member
fruitrow fails with something like this, most recently in [1], other
occurances are [2-9]:

|echo "# +++ regress check in contrib/pg_stat_statements +++" &&
PATH="/home/demo/client-code-REL_19_1/buildroot/REL_19_STABLE/pgsql.build/tmp_install/home/demo/build-farm-19.1/buildroot/REL_19_STABLE/inst/bin:/home/demo/client-code-REL_19_1/buildroot/REL_19_STABLE/pgsql.build/contrib/pg_stat_statements:$PATH"
LD_LIBRARY_PATH="/home/demo/client-code-REL_19_1/buildroot/REL_19_STABLE/pgsql.build/tmp_install/home/demo/build-farm-19.1/buildroot/REL_19_STABLE/inst/lib:$LD_LIBRARY_PATH"
INITDB_TEMPLATE='/home/demo/client-code-REL_19_1/buildroot/REL_19_STABLE/pgsql.build'/tmp_install/initdb-template
../../src/test/regress/pg_regress--temp-instance=./tmp_check --inputdir=. --bindir=
--temp-config=/home/demo/build-farm-19.1/buildroot/tmp/buildfarm-JwzW2U/bfextra.conf  --port=5798 --temp-config
../../contrib/pg_stat_statements/pg_stat_statements.conf--dbname=contrib_regression select dml cursors utility
level_trackingplanning user_activity wal entry_timestamp privileges extended parallel plancache squashing cleanup
oldextversions
|# +++ regress check in contrib/pg_stat_statements +++
|# initializing database system by copying initdb template
|# using temp instance on port 5798 with PID 19255
|ERROR:  invalid page in block 97 of relation "base/4/1255"
|ERROR:  database "contrib_regression" does not exist
|# command failed: "psql" -X -q -c "CREATE DATABASE \\"contrib_regression\\" TEMPLATE=template0" -c "ALTER DATABASE
\\"contrib_regression\\"SET lc_messages TO 'C';ALTER DATABASE \\"contrib_regression\\" SET lc_monetary TO 'C';ALTER
DATABASE\\"contrib_regression\\" SET lc_numeric TO 'C';ALTER DATABASE \\"contrib_regression\\" SET lc_time TO 'C';ALTER
DATABASE\\"contrib_regression\\" SET bytea_output TO 'hex';ALTER DATABASE \\"contrib_regression\\" SET
timezone_abbreviationsTO 'Default';" "postgres" 
|Bail out!make: *** [../../src/makefiles/pgxs.mk:451: check] Fehler 2

The postmaster log for the pg_stat_statements regress check has this:

|2026-10-06 22:29:40.701 BST [19267:4] pg_regress LOG:  statement: CREATE DATABASE "contrib_regression"
TEMPLATE=template0
|2026-10-06 22:29:41.599 BST [19256:1] LOG:  page verification failed, calculated checksum 48592 but expected 64049
|2026-10-06 22:29:41.599 BST [19256:2] CONTEXT:  I/O worker executing I/O on behalf of process 19267
|2026-10-06 22:29:41.600 BST [19256:3] LOG:  invalid page in block 97 of relation "base/4/1255"
|2026-10-06 22:29:41.600 BST [19256:4] CONTEXT:  I/O worker executing I/O on behalf of process 19267
|2026-10-06 22:29:41.674 BST [19267:5] pg_regress ERROR:  invalid page in block 97 of relation "base/4/1255"
|2026-10-06 22:29:41.674 BST [19267:6] pg_regress STATEMENT:  CREATE DATABASE "contrib_regression" TEMPLATE=template0
|2026-10-06 22:29:41.717 BST [19267:7] pg_regress LOG:  statement: ALTER DATABASE "contrib_regression" SET lc_messages
TO'C';ALTER DATABASE "contrib_regression" SET lc_monetary TO 'C';ALTER DATABASE "contrib_regression" SET lc_numeric TO
'C';ALTERDATABASE "contrib_regression" SET lc_time TO 'C';ALTER DATABASE "contrib_regression" SET bytea_output TO
'hex';ALTERDATABASE "contrib_regression" SET timezone_abbreviations TO 'Default'; 
|2026-10-06 22:29:41.718 BST [19267:8] pg_regress ERROR:  database "contrib_regression" does not exist
|2026-10-06 22:29:41.718 BST [19267:9] pg_regress STATEMENT:  ALTER DATABASE "contrib_regression" SET lc_messages TO
'C';ALTERDATABASE "contrib_regression" SET lc_monetary TO 'C';ALTER DATABASE "contrib_regression" SET lc_numeric TO
'C';ALTERDATABASE "contrib_regression" SET lc_time TO 'C';ALTER DATABASE "contrib_regression" SET bytea_output TO
'hex';ALTERDATABASE "contrib_regression" SET timezone_abbreviations TO 'Default'; 
|2026-10-06 22:29:41.724 BST [19267:10] pg_regress LOG:  disconnection: session time: 0:00:01.026 user=demo
database=postgreshost=[local] 
|2026-10-06 22:29:41.734 BST [19255:4] LOG:  received fast shutdown request
|2026-10-06 22:29:41.734 BST [19255:5] LOG:  aborting any active transactions
|2026-10-06 22:29:41.749 BST [19255:6] LOG:  background worker "logical replication launcher" (PID 19263) exited with
exitcode 1 
|2026-10-06 22:29:41.750 BST [19258:1] LOG:  shutting down
|2026-10-06 22:29:41.751 BST [19258:2] LOG:  checkpoint starting: shutdown fast
|2026-10-06 22:29:41.761 BST [19258:3] LOG:  checkpoint complete: shutdown fast: wrote 3 buffers (0.0%), wrote 3 SLRU
buffers;0 WAL file(s) added, 0 removed, 0 recycled; write=0.006 s, sync=0.001 s, total=0.010 s; sync files=0,
longest=0.000s, average=0.000 s; distance=3085 kB, estimate=3085 kB; lsn=0/01A5FE00, redo lsn=0/01A5FE00 
|2026-10-06 22:29:41.795 BST [19255:7] LOG:  database system is shut down

The frequency is low, but so far it only ever hit in 18/19/HEAD and it
always had the 'CONTEXT:  I/O worker executing I/O on behalf of process
[...]' message, which indicates this only happens in AIO.

I since configured the buildfarm member to keep the failing inst/build
tree, but when I run pg_checksums on it, there is no error, so I think
this is not storage corruption, but rather something in-memory:

|$ LANG=C /home/demo/build-farm-19.1/buildroot/REL_19_STABLE/instkeep.2026-10-06_22-14-32/bin/pg_checksums -v -D
/home/demo/build-farm-19.1/buildroot/REL_19_STABLE/pgsqlkeep.2026-10-06_22-14-32/contrib/pg_stat_statements/tmp_check/data/
2>&1| grep 4.1255.$ 
|pg_checksums: checksums verified in file
"/home/demo/build-farm-19.1/buildroot/REL_19_STABLE/pgsqlkeep.2026-10-06_22-14-32/contrib/pg_stat_statements/tmp_check/data//base/4/1255"

I tried to reproduce the problem, but so far had no luck.

Looking at
https://wiki.postgresql.org/wiki/Known_Buildfarm_Test_Failures (and
CC'ing Alexander), it looks like this error has not appeared on any
other buildfarm members, so it looks like it is specific to GNU Hurd.

Anybody have an idea on how to diagnose that and whether this is
interesting to pursue anyway?


Michael

[1] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-10-06%2021%3A14%3A32&stg=misc-check
[2] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-09-22%2013%3A10%3A04&stg=misc-check
[3] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-09-19%2001%3A10%3A03&stg=check
[4] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-09-04%2007%3A29%3A28&stg=misc-check
[5] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-08-19%2003%3A20%3A17&stg=check
[6] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-08-01%2013%3A10%3A03&stg=misc-check
[7] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-07-28%2003%3A24%3A18&stg=misc-check
[8] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-07-01%2014%3A56%3A27&stg=misc-check
[9] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fruitcrow&dt=2026-06-03%2002%3A48%3A51&stg=misc-check