Re: Compression of bigger WAL records - Mailing list pgsql-hackers

From Andrey Borodin
Subject Re: Compression of bigger WAL records
Date
Msg-id B2902A9C-6BFA-493B-8FE9-A3A733DB1CA3@yandex-team.ru
Whole thread
In response to Re: Compression of bigger WAL records  (Michael Paquier <michael@paquier.xyz>)
Responses Re: Compression of bigger WAL records
List pgsql-hackers
Hi Michael!

Thanks for looking into this!

Attached v10 contains only the context-reuse changes, split into replay
(0001) and WAL insertion (0002).

The reader now owns a ZSTD_DCtx pointer under USE_ZSTD.  Backend builds
register a reset callback in the reader's memory context, and
XLogReaderFree() unregisters it before freeing the context.  I also
shortened the compression-side comment.

We could also avoid this cleanup bookkeeping with a static context, as
in 0002. For now we only decompress independent images in one call, with
no history to preserve between records.

On 6 Oct 2026, at 03:34 UTC, Michael Paquier wrote:
> The replay part makes the good choice, so I'm puzzled as of why you
> have chosen a safe failure mode here.

My reasoning was that the writer can still store an uncompressed image,
whereas replay cannot restore a compressed image without the context.
I changed this as you suggested, but I still think falling back to an
uncompressed image is preferable. We are in a critical section.

wal_compression can change during a session, so preparing the workspace
outside critical sections is not straightforward. This is also the
hardest and most contentious part of the whole-record and streaming
compression patches. I'm trying to reuse the FPI compression buffers,
but still need twice as much buffer space as before. I haven't found a
better approach. Your thoughts on the allocation strategy would be very
helpful, even without reviewing those patches.

Thank you!


Best regards, Andrey Borodin.


Attachment

pgsql-hackers by date:

Previous
From: Vaibhav Dalvi
Date:
Subject: Re: remote_apply commit hangs when wal_receiver_status_interval = 0
Next
From: Jakub Wartak
Date:
Subject: Re: enhancing pg_basebackup speeds up to ~23Gbps (small fixes + io_uring/Direct I/O)