From 7fe4a27838c8077c8e0a57e983f96e03e1e89e96 Mon Sep 17 00:00:00 2001 From: Maxim Orlov Date: Fri, 11 Mar 2022 11:35:16 +0300 Subject: [PATCH v53 5/5] README.XID64 Author: Alexander Korotkov Author: Teodor Sigaev Author: Nikita Glukhov Author: Maxim Orlov Author: Pavel Borisov Author: Yura Sokolov Discussion: https://postgr.es/m/CACG%3DezZe1NQSCnfHOr78AtAZxJZeCvxrts0ygrxYwe%3DpyyjVWA%40mail.gmail.com Discussion: https://postgr.es/m/CAJ7c6TPDOYBYrnCAeyndkBktO0WG2xSdYduTF0nxq%2BvfkmTF5Q%40mail.gmail.com --- src/backend/access/heap/README.XID64 | 134 +++++++++++++++++++++++++++ 1 file changed, 134 insertions(+) create mode 100644 src/backend/access/heap/README.XID64 diff --git a/src/backend/access/heap/README.XID64 b/src/backend/access/heap/README.XID64 new file mode 100644 index 0000000000..9b4e6e03a2 --- /dev/null +++ b/src/backend/access/heap/README.XID64 @@ -0,0 +1,134 @@ +src/backend/access/heap/README.XID64 + +64-bit Transaction ID's (XID) +============================= + +A limited number (N = 2^32) of XID's required to do vacuum freeze to prevent +wraparound every N/2 transactions. This causes performance degradation due +to the need to read and rewrite all not yet frozen pages tables while being +vacuumed. In each wraparound cycle, SLRU buffers are also being cut. + +With 64-bit XID's wraparound is effectively postponed to a very distant +future. Even in highly loaded systems that had 2^32 transactions per day +it will take huge 2^31 days before the first enforced "vacuum to prevent +wraparound"). Buffers cutting and routine vacuum are not enforced, and DBA +can plan them independently at the time with the least system load and least +critical for database performance. Also, it can be done less frequently +(several times a year vs every several days) on systems with transaction rates +similar to those mentioned above. + +On-disk tuple and page format +----------------------------- + +On-disk tuple format remains unchanged. 32-bit t_xmin and t_xmax store the +lower parts of 64-bit XMIN and XMAX values. Each heap page has additional +64-bit pd_xid_base and pd_multi_base which are common for all tuples on a page. +They are placed into a pd_special area - 16 bytes in the end of a heap page. +Actual XMIN/XMAX for a tuple are calculated upon reading a tuple from a page +as follows: + +XMIN = t_xmin + pd_xid_base. (1) +XMAX = t_xmax + pd_xid_base/pd_multi_base. (2) + +"Double XMAX" page format +--------------------------------- + +At first read of a heap page after pg_upgrade from 32-bit XID PostgreSQL +version pd_special area with a size of 16 bytes should be added to a page. +Though a page may not have space for this. Then it can be converted to a +temporary format called "double XMAX". + +All tuples after pg-upgrade doesn't need t_xmin anymore as no older transactions +could be running. So we don't need tuple header t_xmin field and we reuse +t_xmin to store higher 32 bits of its XMAX. + +Double XMAX format is only for full pages that don't have 16 bytes for +pd_special. So it neither has a place for a single tuple. Insert and HOT update +for double XMAX pages is impossible and not supported. We can only read or +delete tuples from it. + +When we are able to prune page double XMAX it will be converted from it to +general 64-bit XID page format with all operations on its tuples supported. + +In-memory tuple format +---------------------- + +In-memory tuple representation consists of two parts: +- HeapTupleHeader from disk page (contains all heap tuple contents, not only +header) +- HeapTuple with additional in-memory fields + +HeapTuple for each tuple in memory stores 64bit XMIN/XMAX. They are +precalculated on tuple read from page with (1) and (2). + +The filling of XMIN and XMAX in HeapTuple is done in the same way as the other +fields of HeapTuple struct. It is done in all cases of HeapTuple manipulation. + +Update/delete with 64-bit XIDs and 32-bit t_xmin/t_xmax +-------------------------------------------------------------- + +When we try to delete/update a tuple, we check that XMAX for a page fits (2). +I.e. that t_xmax will not be over MaxShortTransactionId relative to +pd_xid_base/pd_multi_base of a its page. + +If the current XID doesn't fit a range +(pd_xid_base, pd_xid_base + MaxShortTransactionId) (3): + +- heap_page_prepare_for_xid() will try to increase pd_xid_base/pd_multi_base on +a page and update all t_xmin/t_xmax of the other tuples on the page to +correspond new pd_xid_base/pd_multi_base. + +- If it was impossible, it will try to prune and freeze tuples on a page. + +- If this is unsuccessful it will throw an error. Normally this is very +unlikely but if there is a very old living transaction with an age of around +2^32 this can arise. Basically, this is a behavior similar to one during the +vacuum to prevent wraparound when XID was 32-bit. DBA should take care and +avoid very-long-living transactions with an age close to 2^32. So long-living +transactions often they are most likely defunct. + +Insert with 64-bit XIDs and 32-bit t_xmin/t_xmax +------------------------------------------------ + +On insert we check if current XID fits a range (3). Otherwise: + +- heap_page_prepare_for_xid() will try to increase pd_xid_base for t_xmin will +not be over MaxShortTransactionId. + +- If it is impossible, then it will try to prune and freeze tuples on a page. + +Known issue: if pd_xid_base could not be shifted to accommodate a tuple being +inserted due to a very long-running transaction, we just throw an error. We +neither try to insert a tuple into another page nor mark the current page as +full. So, in this (unlikely) case we will get regular insert errors on the next +tries to insert to the page 'locked' by this very long-running transaction. + +Upgrade from 32-bit XID versions +-------------------------------- + +pg_upgrade doesn't change pages format itself. It is done lazily after. + +1. At first heap page read, tuples on a page are repacked to free 16 bytes +at the end of a page, possibly freeing space from dead tuples. + +2A. 16 bytes of pd_special is added if there is a place for it + +2B. Page is converted to "Double XMAX" format if there is no place for +pd_special + +3. If a page is in double XMAX format after its first read, and vacuum (or +micro-vacuum at select query) could prune some tuples and free space for +pd_special, prune_page will add pd_special and convert page from double XMAX +to general 64-bit XID page format. + +This lazy conversion is called only on pages being read. This can slow down +performance after upgrade, but just for a short period of time while "hot" +pages are read (and therefore converted to 64-bit format). + +There is a special case when the first read of a tuple is done in read-only +state (in read-only transaction or on replica). This tuples are to be converted +"in memory", but not sync "to disk", unless cluster or transaction changed to +read-write state (e.g. replica is promoted). In order to support this, we mark +"in memory" pages with converted tuples with bit REGBUF_CONVERTED in buffer +descriptor. When in read-write state this will trigger full page write xlog +record. -- 2.45.2