O(n^2) system calls in RemoveOldXlogFiles() - Mailing list pgsql-hackers

From Thomas Munro
Subject O(n^2) system calls in RemoveOldXlogFiles()
Date
Msg-id CA+hUKG+DRiF9z1_MU4fWq+RfJMxP7zjoptfcmuCFPeO4JM2iVg@mail.gmail.com
Whole thread Raw
Responses Re: O(n^2) system calls in RemoveOldXlogFiles()  (Andres Freund <andres@anarazel.de>)
Re: O(n^2) system calls in RemoveOldXlogFiles()  (Alvaro Herrera <alvherre@alvh.no-ip.org>)
List pgsql-hackers
Hi,

I noticed that RemoveXlogFile() has this code:

        /*
         * Before deleting the file, see if it can be recycled as a future log
         * segment. Only recycle normal files, pg_standby for example can create
         * symbolic links pointing to a separate archive directory.
         */
        if (wal_recycle &&
                endlogSegNo <= recycleSegNo &&
                lstat(path, &statbuf) == 0 && S_ISREG(statbuf.st_mode) &&
                InstallXLogFileSegment(&endlogSegNo, path,
                                                           true,
recycleSegNo, true))
        {
                ereport(DEBUG2,
                                (errmsg("recycled write-ahead log file \"%s\"",
                                                segname)));
                CheckpointStats.ckpt_segs_recycled++;
                /* Needn't recheck that slot on future iterations */
                endlogSegNo++;
        }

I didn't check the migration history of this code but it seems that
endlogSegNo doesn't currently have the right scoping to achieve the
goal of that last comment, so checkpoints finish up repeatedly search
for the next free slot, starting at the low end each time, like so:

stat("pg_wal/00000001000000000000004F", {st_mode=S_IFREG|0600,
st_size=16777216, ...}) = 0
...
stat("pg_wal/000000010000000000000073", 0x7fff98b9e060) = -1 ENOENT
(No such file or directory)

stat("pg_wal/00000001000000000000004F", {st_mode=S_IFREG|0600,
st_size=16777216, ...}) = 0
...
stat("pg_wal/000000010000000000000074", 0x7fff98b9e060) = -1 ENOENT
(No such file or directory)

... and so on until we've recycled all our recyclable segments.  Ouch.



pgsql-hackers by date:

Previous
From: Bharath Rupireddy
Date:
Subject: Re: Parallel Inserts in CREATE TABLE AS
Next
From: "Andrey V. Lepikhov"
Date:
Subject: Re: Removing unneeded self joins