Re: WAL segment file descriptor leak on read errors can PANIC the server - Mailing list pgsql-hackers

> On Sep 22, 2026, at 16:15, Chao Li <li.evan.chao@gmail.com> wrote:
>
>
>
>> On Sep 21, 2026, at 15:12, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote:
>>
>> Hi,
>>
>> A failed WAL read leaks the open WAL segment file descriptor until the
>> backend exits. Once the fd limit is reached, all later queries in that
>> backend fail with "Too many open files". With a connection pooler the
>> leaks add up across clients. There is no warning or log message for
>> this. Reproducers are at [1] and [2].
>>
>> It can happen with logical decoding that cancelled decode calls
>> exhaust the fds, and depending on which open fails first, the backend
>> dies with a FATAL or the instance restarts with a PANIC. [2] shows a
>> PANIC.
>>
>> It can also happen that the leaked fd is on a WAL segment that has
>> since been removed. The file is unlinked but its space is not freed
>> until the backend goes away, so pg_wal still looks small while free
>> space on the disk keeps shrinking. On a small WAL volume I could fill
>> the disk this way, and the server then PANICed while writing a new WAL
>> file with no space left, which took down the database instance.
>>
>> I noticed this while looking at the error paths in pg_walinspect for
>> the reported issue [3]. I then used Claude Code to check for the same
>> issue elsewhere, and it helped me find the other affected paths.
>>
>> The WAL read paths open the segment file as a plain kernel fd. It is
>> not a virtual fd and not a transient file, and no resource owner owns
>> it, so nothing in the backend knows it is open. On an error, the
>> memory holding the WAL reader is freed with its context, but the
>> segment file it had open stays open. This leak seems to exist in
>> pg_walinspect functions, logical decoding functions, 2PC WAL read
>> code, and the WAL summarizer. All of these except the WAL summarizer
>> are reachable from SQL in simple ways. The walsender does not have
>> this issue because it closes the file in its own error cleanup. The
>> startup process is fine too, its reader does not keep a segment open
>> this way, and its read errors are FATAL anyway.
>>
>> This is the same leak that commit 91c40548d5 fixed for 2PC by closing
>> the fd in XLogReaderFree(), but that close only runs on the normal
>> path, not when an error is thrown before it.
>>
>> I think the fix is to register a memory context reset callback on the
>> context the reader is allocated in. If that context is reset or
>> deleted while a segment file is still open, the callback closes it.
>> Doing this in XLogReaderAllocate() covers every caller, present and
>> future, instead of adding an error handler to each one. On HEAD,
>> XLogReaderFree() unregisters the callback and the reader keeps a
>> pointer to it in a new field. On PG18 and older the reader struct
>> cannot grow because of ABI, and there is no
>> MemoryContextUnregisterResetCallback(), so there the callback stays
>> registered and its state lives in a small list local to the WAL
>> reader. I attached a nocfbot prefixed patch for the back branches
>> doing that.
>>
>> There are alternative approaches that have some issues. Wrapping each
>> caller in its own error handler works but is easy to miss in future
>> callers. Opening the segment as a transient file, so that it gets
>> closed at transaction abort, adds a rule that a reader cannot outlive
>> a transaction, which does not work for the logical walsender since it
>> reads WAL outside any transaction. Tracking the fd with a resource
>> owner also fixes all callers in one place, but the extensible resource
>> owner API only exists in PG17 and later, so it cannot be used in the
>> older branches.
>>
>> Please find the attached patches (v1 for HEAD and PG19, nocfbot for
>> PG18). If the approach looks good, I will prepare patches for all the
>> remaining back branches using the PG18 approach.
>>
>> Thoughts?
>>
>> [1]
>> $ ulimit -n 200
>>
>> CREATE TABLE t1 (a int);
>> INSERT INTO t1 VALUES (1);
>>
>> -- Each iteration errors at end of WAL and leaks one fd
>> DO $$
>> BEGIN
>> FOR i IN 1..200 LOOP
>>   BEGIN
>>     PERFORM * FROM
>> pg_get_wal_records_info(pg_current_wal_flush_lsn(),
>> 'FFFFFFFF/FFFFFFFF');
>>   EXCEPTION WHEN OTHERS THEN
>>     NULL;
>>   END;
>> END LOOP;
>> END $$;
>>
>> -- Session can no longer open any file
>> postgres=# SELECT count(*) FROM t1;
>> ERROR:  could not open file "base/5/2691": Too many open files
>> LINE 1: SELECT count(*) FROM t1;
>>              ^
>> postgres=# CREATE TABLE t2 (a int);
>> ERROR:  could not create file "base/5/16398": Too many open files
>>
>> -- Shows the fd table full of pg_wal segments
>> $ lsof -w -p <backend pid> | grep -c pg_wal
>> 193
>>
>> [2]
>> $ ulimit -n 200
>>
>> SELECT pg_create_logical_replication_slot('s', 'test_decoding');
>> CREATE TABLE foo(a int, b text);
>> INSERT INTO foo SELECT i, repeat('x', 200) FROM generate_series(1, 300000) i;
>>
>> -- Each cancelled decode errors mid-read and leaks one fd
>> SET statement_timeout = '10ms';
>> SELECT 'SELECT * FROM pg_logical_slot_peek_changes(''s'', NULL, NULL);'
>> FROM generate_series(1, 200) \gexec
>>
>> ERROR:  canceling statement due to statement timeout
>> ERROR:  canceling statement due to statement timeout
>> ..
>> ..
>> ERROR:  canceling statement due to statement timeout
>> ERROR:  canceling statement due to statement timeout
>> PANIC:  could not open file "pg_logical/snapshots/0-18692A8.snap": Too
>> many open files
>> server closed the connection unexpectedly
>>       This probably means the server terminated abnormally
>>       before or while processing the request.
>> The connection to the server was lost. Attempting reset: Failed.
>>
>> [3] https://postgr.es/m/80E9F0AD-CFC5-4BE5-81DE-D8FE35E10A1C@gmail.com
>>
>> --
>> Bharath Rupireddy
>> Amazon Web Services: https://aws.amazon.com
>>
<v1-0001-Fix-WAL-segment-file-descriptor-leak-on-WAL-read-.patch><nocfbot-v1-0001-PG18-Fix-WAL-segment-file-descriptor-leak.patch>
>
> I have objection on releasing the FD by a callback. A few comments on v1:
>

I just noticed this stupid typo. I meant to say “NO objection”.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/