Thread: zstd failing on mipsel (PG 15.12, pg_verifybackup/t/010_client_untar.pl)

I uploaded 15.12 to Debian bookworm (stable), but the mipsel (only
that) build is consistently failing, while 15.11 was working there
just two weeks earlier. No commit between 15.11 and 15.12 looks
remotely related, and zstd in bookworm did not change.

https://buildd.debian.org/status/logs.php?pkg=postgresql-15&arch=mipsel

Bailout called.  Further testing stopped:  command "/usr/bin/zstd -d
/build/reproducible-path/postgresql-15-15.12/build/src/bin/pg_verifybackup/tmp_check/t_010_client_untar_primary_data/backup/client-backup/base.tar.zst"
exitedwith value 37
 
t/010_client_untar.pl ..
ok 1 - client side backup, compression none
ok 2 - found expected backup files, compression none
ok 3 - verify backup, compression none
ok 4 - client side backup, compression gzip
ok 5 - found expected backup files, compression gzip
ok 6 - verify backup, compression gzip
ok 7 - client side backup, compression lz4
ok 8 - found expected backup files, compression lz4
ok 9 - verify backup, compression lz4
ok 10 - client side backup, compression zstd
ok 11 - found expected backup files, compression zstd
ok 12 - verify backup, compression zstd
ok 13 - client side backup, compression parallel zstd
ok 14 - found expected backup files, compression parallel zstd
Dubious, test returned 255 (wstat 65280, 0xff00)
All 14 subtests passed

# Running: /usr/bin/zstd -d
/build/reproducible-path/postgresql-15-15.12/build/src/bin/pg_verifybackup/tmp_check/t_010_client_untar_primary_data/backup/client-backup/base.tar.zst
zstd: error 37 : Read error
[00:13:47.308](0.006s) Bail out!  command "/usr/bin/zstd -d
/build/reproducible-path/postgresql-15-15.12/build/src/bin/pg_verifybackup/tmp_check/t_010_client_untar_primary_data/backup/client-backup/base.tar.zst"
exitedwith value 37
 

Error 37 from zstd is this:

/* AIO_ReadPool_executeReadJob:
 * Executes a read job synchronously. Can be used as a function for a thread pool. */
static void AIO_ReadPool_executeReadJob(void* opaque){
    IOJob_t* const job = (IOJob_t*) opaque;
    ReadPoolCtx_t* const ctx = (ReadPoolCtx_t *)job->ctx;
    if(ctx->reachedEof) {
        job->usedBufferSize = 0;
        AIO_ReadPool_addJobToCompleted(job);
        return;
    }
    job->usedBufferSize = fread(job->buffer, 1, job->bufferSize, job->file);
    if(job->usedBufferSize < job->bufferSize) {
        if(ferror(job->file)) {
            EXM_THROW(37, "Read error");
        } else if(feof(job->file)) {
            ctx->reachedEof = 1;
        } else {
            EXM_THROW(37, "Unexpected short read");
        }
    }
    AIO_ReadPool_addJobToCompleted(job);
}

I cannot reproduce the problem on the Debian mipsel porter box, and
the (working) test does not leave the base.tar.zst mentioned about
behind.

So I have no idea how to proceed here. Does any of this ring a bell?
Are we hitting a zstd bug like https://github.com/facebook/zstd/pull/3517
(but this one is claimed to be fixed with a patch in the bookworm
package)?

Christoph

Re: To PostgreSQL Hackers
> I cannot reproduce the problem on the Debian mipsel porter box, and
> the (working) test does not leave the base.tar.zst mentioned about
> behind.

Since I don't have access to the machine where it is failing, I
uploaded a new version that base64-encodes the "bad" base.tar.zst file
and spits it into the build log:

https://buildd.debian.org/status/fetch.php?pkg=postgresql-15&arch=mipsel&ver=15.12-0%2Bdeb12u2&stamp=1741278690&raw=0

... and the file is just fine. The blame is now on the kernel side for
any machine-level problems. (The kernel on the porter box where I
tried to reproduce the problem is much older than on the actual
buildds. A new version that hopefully fixes the problem is being built
now.)

So, nothing to see on the PG side. Thanks.

Christoph