FreezeLimit underflows in pg14 and 15 causing incorrect behavior in heap_prepare_freeze_tuple - Mailing list pgsql-hackers

While investigating a bug over on [1], I found that
vacuum_set_xid_limits() is calculating freezeLimit in an unsafe way on
at least Postgres 14 and 15.

    limit = *oldestXmin - freezemin;
    safeLimit = ReadNextTransactionId() - autovacuum_freeze_max_age;
    if (TransactionIdPrecedes(limit, safeLimit))
        limit = *oldestXmin;
    *freezeLimit = limit;

All of these are unsigned, so it doesn't work very nicely when
freezemin (based on autovacuum_freeze_min_age) is larger than
oldestXmin and autovacuum_freeze_max_age is bigger than the next
transaction ID -- which is pretty common right after initdb, for
example.

I noticed the effect of this because FreezeLimit is saved in the
LVRelState and passed to heap_prepare_freeze_tuple() as cutoff_xid,
which is used to guard against freezing tuples that shouldn't be
frozen.

I didn't propose a fix because I just want to make sure I'm not
missing something first.

- Melanie

[1] https://www.postgresql.org/message-id/CAAKRu_Y_NJzF4-8gzTTeaOuUL3CcGoXPjXcAHbTTygT8AyVqag%40mail.gmail.com