Thread: pg_serial early wraparound
Hi hackers, The SLRU managed by predicate.c can wrap around and overwrite data if you have more than 1 billion active XIDs. That's because when SSI was implemented, slru.c was limited to four digit segment names, which implied a page limit that wasn't enough for pg_serial to have space for every possible XID. We should probably rip that code out, because SLRUs now support five digit segment names. Something like the attached. I'll post a test script to demonstrate correct wraparound behaviour around in time for one of the later CFs. -- Thomas Munro http://www.enterprisedb.com
Attachment
On Wed, Nov 9, 2016 at 11:07 AM, Thomas Munro <thomas.munro@enterprisedb.com> wrote: > The SLRU managed by predicate.c can wrap around and overwrite data if > you have more than 1 billion active XIDs. That's because when SSI was > implemented, slru.c was limited to four digit segment names, which > implied a page limit that wasn't enough for pg_serial to have space > for every possible XID. We should probably rip that code out, because > SLRUs now support five digit segment names. Something like the > attached. I'll post a test script to demonstrate correct wraparound > behaviour around in time for one of the later CFs. Here is a shell script that shows a full rotation through xid space if you build PostgreSQL with TEST_OLDSERXID, which you can do by uncommenting a line in predicate.c. On master we see the SLRU segments go around the clock twice for each time xid goes around. With the patch it goes around just once, adding an extra character to the segment name to double the space. By the way, I think the real number of xids it can hold today is (65536 * 32 * 8192) / sizeof(uint64) = 2^31 xids, not 2^30 as indicated by an existing comment. So I think there is actually enough space and wrapping is probably harmess, but it seems cleaner and simpler not to do that and to rip out the scary warning code, so I'll add this to the CF. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Attachment
On Mon, Feb 27, 2017 at 7:28 PM, Thomas Munro <thomas.munro@enterprisedb.com> wrote: > On Wed, Nov 9, 2016 at 11:07 AM, Thomas Munro > <thomas.munro@enterprisedb.com> wrote: >> The SLRU managed by predicate.c can wrap around and overwrite data if >> you have more than 1 billion active XIDs. That's because when SSI was >> implemented, slru.c was limited to four digit segment names, which >> implied a page limit that wasn't enough for pg_serial to have space >> for every possible XID. We should probably rip that code out, because >> SLRUs now support five digit segment names. Something like the >> attached. I'll post a test script to demonstrate correct wraparound >> behaviour around in time for one of the later CFs. > > Here is a shell script that shows a full rotation through xid space if > you build PostgreSQL with TEST_OLDSERXID, which you can do by > uncommenting a line in predicate.c. On master we see the SLRU > segments go around the clock twice for each time xid goes around. > With the patch it goes around just once, adding an extra character to > the segment name to double the space. I attached the wrong version. Here is the right one. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Attachment
The following review has been posted through the commitfest application:
make installcheck-world: not tested
Implements feature: not tested
Spec compliant: not tested
Documentation: not tested
Hi, I've tried to review this patch, but it seems that I miss something essential.
You claim that SLRUs now support five digit segment name, while in slru.h
at current master I see the following:
* Note: slru.c currently assumes that segment file names will be four hex* digits. This sets a lower bound on the
segmentsize (64K transactions* for 32-bit TransactionIds).*/
#define SLRU_PAGES_PER_SEGMENT 32
/* Maximum length of an SLRU name */
#define SLRU_MAX_NAME_LENGTH 32
Could you please clarify the idea of the patch? Is it still relevant?
I've also run your test script.
pg_clog was renamed to pg_xact, so it need to be changed accordingly
echo "Contents of pg_clog:" ls $PGDATA/pg_xact/
The test shows failed assertion:
========== setting next xid to 1073741824 =========
Transaction log reset
waiting for server to start....2017-03-24 17:05:19.897 MSK [1181] LOG: listening on IPv4 address "127.0.0.1", port
5432
2017-03-24 17:05:19.981 MSK [1181] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2017-03-24 17:05:20.081 MSK [1183] LOG: database system was shut down at 2017-03-24 17:05:19 MSK
2017-03-24 17:05:20.221 MSK [1181] LOG: database system is ready to accept connectionsdone
server started
vacuumdb: vacuuming database "postgres"
vacuumdb: vacuuming database "template0"
vacuumdb: vacuuming database "template1"
TRAP: FailedAssertion("!(TransactionIdPrecedesOrEquals(oldestXact, ShmemVariableCache->oldestXid))", File: "clog.c",
Line:669)
vacuumdb: vacuuming of database "template1" failed: server closed the connection unexpectedly This probably means the
serverterminated abnormally before or while processing the request.
2017-03-24 17:05:21.541 MSK [1181] LOG: server process (PID 1202) was terminated by signal 6: Aborted
2017-03-24 17:05:21.541 MSK [1181] DETAIL: Failed process was running: VACUUM (FREEZE);
The new status of this patch is: Waiting on Author
On Sat, Mar 25, 2017 at 3:11 AM, Anastasia Lubennikova
<lubennikovaav@gmail.com> wrote:
> Hi, I've tried to review this patch, but it seems that I miss something essential.
Hi Anastasia,
Thanks for looking at this.
> You claim that SLRUs now support five digit segment name, while in slru.h
> at current master I see the following:
>
> * Note: slru.c currently assumes that segment file names will be four hex
> * digits. This sets a lower bound on the segment size (64K transactions
> * for 32-bit TransactionIds).
> */
> #define SLRU_PAGES_PER_SEGMENT 32
>
> /* Maximum length of an SLRU name */
> #define SLRU_MAX_NAME_LENGTH 32
That comment is out of date. Commit 638cf09e extended SLRUs to
support 5 digit names, to support pg_multixact. And I see now that
commit 73c986ad more recently created the possibility of 6 chacater
SLRU file names for pg_commit_ts.
> Could you please clarify the idea of the patch? Is it still relevant?
The idea is simply to remove some strange old code including scary
error messages that is no longer needed. In my study of predicate.c
for other reasons, I noticed this in passing and thought I'd tidy it
up. Because I have tangled with pg_multixact and seen 5-character
SLRU files with my own eyes, I knew that the restriction that
motivated this code was no longer valid.
> I've also run your test script.
> pg_clog was renamed to pg_xact, so it need to be changed accordingly
> echo "Contents of pg_clog:"
> ls $PGDATA/pg_xact/
Right.
> The test shows failed assertion:
>
> ========== setting next xid to 1073741824 =========
> Transaction log reset
> waiting for server to start....2017-03-24 17:05:19.897 MSK [1181] LOG: listening on IPv4 address "127.0.0.1", port
5432
> 2017-03-24 17:05:19.981 MSK [1181] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
> 2017-03-24 17:05:20.081 MSK [1183] LOG: database system was shut down at 2017-03-24 17:05:19 MSK
> 2017-03-24 17:05:20.221 MSK [1181] LOG: database system is ready to accept connections
> done
> server started
> vacuumdb: vacuuming database "postgres"
> vacuumdb: vacuuming database "template0"
> vacuumdb: vacuuming database "template1"
> TRAP: FailedAssertion("!(TransactionIdPrecedesOrEquals(oldestXact, ShmemVariableCache->oldestXid))", File: "clog.c",
Line:669)
> vacuumdb: vacuuming of database "template1" failed: server closed the connection unexpectedly
> This probably means the server terminated abnormally
> before or while processing the request.
> 2017-03-24 17:05:21.541 MSK [1181] LOG: server process (PID 1202) was terminated by signal 6: Aborted
> 2017-03-24 17:05:21.541 MSK [1181] DETAIL: Failed process was running: VACUUM (FREEZE);
My cheap trick for moving the xid around the clock quickly to test
wraparound scenarios no longer works, since this new assertion was
added in ea42cc18. That was committed just a few hours before you
tested this. Bad luck for me!
> The new status of this patch is: Waiting on Author
It's not urgent, it's just cleanup work, so I've now moved it to the
next commitfest. I will try to figure out a new way to demonstrate
that it works correctly without having to ask a reviewing to disable
any assertions. Thanks again.
--
Thomas Munro
http://www.enterprisedb.com
On Sat, Mar 25, 2017 at 7:27 AM, Thomas Munro <thomas.munro@enterprisedb.com> wrote: > On Sat, Mar 25, 2017 at 3:11 AM, Anastasia Lubennikova > <lubennikovaav@gmail.com> wrote: >> You claim that SLRUs now support five digit segment name, while in slru.h >> at current master I see the following: >> >> * Note: slru.c currently assumes that segment file names will be four hex >> * digits. This sets a lower bound on the segment size (64K transactions >> * for 32-bit TransactionIds). >> */ I've now complained about that comment in a separate thread. > It's not urgent, it's just cleanup work, so I've now moved it to the > next commitfest. I will try to figure out a new way to demonstrate > that it works correctly without having to ask a review[er] to disable > any assertions. Thanks again. Here's a rebased batch. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Attachment
On Wed, Jun 28, 2017 at 1:11 PM, Thomas Munro <thomas.munro@enterprisedb.com> wrote: > On Sat, Mar 25, 2017 at 7:27 AM, Thomas Munro > <thomas.munro@enterprisedb.com> wrote: >> On Sat, Mar 25, 2017 at 3:11 AM, Anastasia Lubennikova >> <lubennikovaav@gmail.com> wrote: >>> You claim that SLRUs now support five digit segment name, while in slru.h >>> at current master I see the following: >>> >>> * Note: slru.c currently assumes that segment file names will be four hex >>> * digits. This sets a lower bound on the segment size (64K transactions >>> * for 32-bit TransactionIds). >>> */ > > I've now complained about that comment in a separate thread. > >> It's not urgent, it's just cleanup work, so I've now moved it to the >> next commitfest. I will try to figure out a new way to demonstrate >> that it works correctly without having to ask a review[er] to disable >> any assertions. Thanks again. Rebased again, now with a commit message. That assertion has since been removed (commit ec99dd5a) so the attached test script can once again be used to see the contents of pg_serial as the xid goes all the way around, if you build with TEST_OLDSERXID defined so that predicate.c forces information about xids out to pg_serial. -- Thomas Munro http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Attachment
On Fri, Sep 1, 2017 at 8:12 PM, Thomas Munro <thomas.munro@enterprisedb.com> wrote: > Rebased again, now with a commit message. That assertion has since > been removed (commit ec99dd5a) so the attached test script can once > again be used to see the contents of pg_serial as the xid goes all the > way around, if you build with TEST_OLDSERXID defined so that > predicate.c forces information about xids out to pg_serial. Moved to next CF per lack of reviews. -- Michael
Greetings, * Thomas Munro (thomas.munro@enterprisedb.com) wrote: > On Wed, Jun 28, 2017 at 1:11 PM, Thomas Munro > <thomas.munro@enterprisedb.com> wrote: > > On Sat, Mar 25, 2017 at 7:27 AM, Thomas Munro > > <thomas.munro@enterprisedb.com> wrote: > >> On Sat, Mar 25, 2017 at 3:11 AM, Anastasia Lubennikova > >> <lubennikovaav@gmail.com> wrote: > >>> You claim that SLRUs now support five digit segment name, while in slru.h > >>> at current master I see the following: > >>> > >>> * Note: slru.c currently assumes that segment file names will be four hex > >>> * digits. This sets a lower bound on the segment size (64K transactions > >>> * for 32-bit TransactionIds). > >>> */ > > > > I've now complained about that comment in a separate thread. > > > >> It's not urgent, it's just cleanup work, so I've now moved it to the > >> next commitfest. I will try to figure out a new way to demonstrate > >> that it works correctly without having to ask a review[er] to disable > >> any assertions. Thanks again. > > Rebased again, now with a commit message. That assertion has since > been removed (commit ec99dd5a) so the attached test script can once > again be used to see the contents of pg_serial as the xid goes all the > way around, if you build with TEST_OLDSERXID defined so that > predicate.c forces information about xids out to pg_serial. I've taken a look through this and it seems pretty reasonable. Would be great to have someone actually try to duplicate the testing that Thomas did (though I have little doubt that it works as described) and get it to Ready-For-Committer state. Anastasia, thanks for the previous review, any chance you could try again with the latest patch (against the current state of git)? Thanks! Stephen
Attachment
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation: not tested The patch doesn't break anything in regression tests and does the code cleanup. As far as I understand, the removed code was dead, since SLRU size is large enough and the wraparound, described in the message is impossible. So I mark it as Ready For Committer. I didn't manage to repeat the attached test, though. Server doesn't start after xid reset. It throws an error: server stopped ========== setting next xid to 1073741824 ========= Write-ahead log reset waiting for server to start....2018-03-12 14:18:59.551 MSK [16126] LOG: listening on IPv4 address "127.0.0.1", port 5432 2018-03-12 14:18:59.625 MSK [16126] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432" 2018-03-12 14:18:59.764 MSK [16127] LOG: database system was shut down at 2018-03-12 14:18:59 MSK 2018-03-12 14:18:59.802 MSK [16127] FATAL: could not access status of transaction 10737418 2018-03-12 14:18:59.802 MSK [16127] DETAIL: Could not open file "pg_xact/000A": Нет такого файла или каталога. 2018-03-12 14:18:59.803 MSK [16126] LOG: startup process (PID 16127) exited with exit code 1 2018-03-12 14:18:59.803 MSK [16126] LOG: aborting startup due to startup process failure 2018-03-12 14:18:59.804 MSK [16126] LOG: database system is shut down
Thomas Munro <thomas.munro@enterprisedb.com> writes:
> Rebased again, now with a commit message. That assertion has since
> been removed (commit ec99dd5a) so the attached test script can once
> again be used to see the contents of pg_serial as the xid goes all the
> way around, if you build with TEST_OLDSERXID defined so that
> predicate.c forces information about xids out to pg_serial.
Couple thoughts here ---
Seems like if the patch is correct as-is, then the OldSerXidPage
macro could be simplified, as the modulo no longer does anything.
Also, OldSerXidSegment doesn't seem to be used.
I'm a little worried because Anastasia couldn't repeat the test;
why is that?
regards, tom lane
On Tue, Mar 27, 2018 at 5:50 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Thomas Munro <thomas.munro@enterprisedb.com> writes: >> Rebased again, now with a commit message. That assertion has since >> been removed (commit ec99dd5a) so the attached test script can once >> again be used to see the contents of pg_serial as the xid goes all the >> way around, if you build with TEST_OLDSERXID defined so that >> predicate.c forces information about xids out to pg_serial. > > Couple thoughts here --- Thanks for looking at this! > Seems like if the patch is correct as-is, then the OldSerXidPage > macro could be simplified, as the modulo no longer does anything. The patch already did that: -#define OldSerXidPage(xid) ((((uint32) (xid)) / OLDSERXID_ENTRIESPERPAGE) % (OLDSERXID_MAX_PAGE + 1)) +#define OldSerXidPage(xid) (((uint32) (xid)) / OLDSERXID_ENTRIESPERPAGE) > Also, OldSerXidSegment doesn't seem to be used. Right, thanks. Removed. > I'm a little worried because Anastasia couldn't repeat the test; > why is that? Hmm. I'm not sure. It works for me on a couple of machines and what I see is: ========== setting next xid to 65536 ========= ... Contents of pg_serial: 0002 ========== setting next xid to 1073741824 ========= ... Contents of pg_serial: 8000 ========== setting next xid to 2147483648 ========= ... Contents of pg_serial: 10000 ========== setting next xid to 3221225472 ========= ... Contents of pg_serial: 18000 ========== setting next xid to 65536 ========= ... Contents of pg_serial: 0002 ========== setting next xid to 1073741824 ========= ... Contents of pg_serial: 8000 -- Thomas Munro http://www.enterprisedb.com
Attachment
Thomas Munro <thomas.munro@enterprisedb.com> writes:
> On Tue, Mar 27, 2018 at 5:50 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I'm a little worried because Anastasia couldn't repeat the test;
>> why is that?
> Hmm. I'm not sure. It works for me on a couple of machines and what I see is:
Yeah, it works for me too, so not sure what the problem was.
The only other point I can think of is that by changing the storage
layout under pg_serial/, this'd create an issue for pg_upgrade, if
pg_upgrade tried to migrate this data. But it doesn't.
Hence, pushed.
regards, tom lane