From 36666e87fe05ed24e413d1cd2f512bfacd65be38 Mon Sep 17 00:00:00 2001 From: Peter Geoghegan Date: Sat, 22 Apr 2023 13:04:13 -0700 Subject: [PATCH v4 9/9] Overhaul freezing and wraparound docs. This is almost a complete rewrite. "Preventing Transaction ID Wraparound Failures" becomes "Freezing to manage the transaction ID space". This is follow-up work to commit 1de58df4, which added page-level freezing to VACUUM. The emphasis is now on the physical work of freezing pages. This flows a little better than it otherwise would due to recent structural cleanups to maintenance.sgml; discussion about freezing now immediately follows discussion of cleanup of dead tuples. We still talk about the problem of the system activating xidStopLimit protections in the same section, but we use much less alarmist language about data corruption, and are no longer overly concerned about the very worst case. We don't rescind the recommendation that users recover from an xidStopLimit outage by using single user mode, though that seems like something we should aim to do in the near future. There is no longer a separate sect3 to discuss MultiXactId related issues. VACUUM now performs exactly the same processing steps when it freezes a page, independent of the trigger condition. Also move recommendation about setting autovacuum_freeze_min_age reloption in append-only tables that was originally added by the autovacuum_vacuum_insert_scale_factor commit over to "Routine Vacuuming", where it now appears in the form of a "Tip" box. Also describe the page-level freezing FPI optimization added by commit 1de58df4. This is expected to trigger the majority of all freezing with many types of workloads. Also move "table age" monitoring query to monitoring.sgml, though leave behind a couple of forwarding links in maintenance.sgml's discussion of freezing and relfrozenxid advancement. --- doc/src/sgml/catalogs.sgml | 18 +- doc/src/sgml/config.sgml | 79 +- doc/src/sgml/logicaldecoding.sgml | 4 +- doc/src/sgml/maintenance.sgml | 1062 ++++++++++++++++----- doc/src/sgml/monitoring.sgml | 80 +- doc/src/sgml/ref/create_table.sgml | 9 +- doc/src/sgml/ref/prepare_transaction.sgml | 14 +- doc/src/sgml/ref/vacuum.sgml | 14 +- doc/src/sgml/ref/vacuumdb.sgml | 17 +- doc/src/sgml/storage.sgml | 3 +- doc/src/sgml/xact.sgml | 2 +- 11 files changed, 963 insertions(+), 339 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 524084055..7bb123e32 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -2243,8 +2243,9 @@ SCRAM-SHA-256$<iteration count>:&l All transaction IDs before this one have been replaced with a permanent (frozen) transaction ID in this table. This is used to track - whether the table needs to be vacuumed in order to prevent transaction - ID wraparound or to allow pg_xact to be shrunk. Zero + whether the table needs an aggressive VACUUM (see + ) or an anti-wraparound autovacuum + (see ). Zero (InvalidTransactionId) if the relation is not a table. @@ -2256,8 +2257,9 @@ SCRAM-SHA-256$<iteration count>:&l All multixact IDs before this one have been replaced by a transaction ID in this table. This is used to track - whether the table needs to be vacuumed in order to prevent multixact ID - wraparound or to allow pg_multixact to be shrunk. Zero + whether the table needs an aggressive VACUUM (see + ) or an anti-wraparound autovacuum + (see ). Zero (InvalidMultiXactId) if the relation is not a table. @@ -3053,8 +3055,8 @@ SCRAM-SHA-256$<iteration count>:&l All transaction IDs before this one have been replaced with a permanent (frozen) transaction ID in this database. This is used to - track whether the database needs to be vacuumed in order to prevent - transaction ID wraparound or to allow pg_xact to be shrunk. + track whether the database allows pg_xact to be + shrunk (see ). It is the minimum of the per-table pg_class.relfrozenxid values. @@ -3067,8 +3069,8 @@ SCRAM-SHA-256$<iteration count>:&l All multixact IDs before this one have been replaced with a transaction ID in this database. This is used to - track whether the database needs to be vacuumed in order to prevent - multixact ID wraparound or to allow pg_multixact to be shrunk. + track whether the database allows pg_multixact to be + shrunk (see ). It is the minimum of the per-table pg_class.relminmxid values. diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 909a3f28c..0a94ff46a 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -2812,7 +2812,9 @@ include_dir 'conf.d' 1min) are only allowed because they may sometimes be useful for testing. While a setting as high as 60d is allowed, please note that in many workloads extreme bloat or - transaction ID wraparound may occur in much shorter time frames. + transaction ID exhaustion may occur in much shorter time frames + (see and + ). @@ -8358,9 +8360,8 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; Note that even when this parameter is disabled, the system - will launch autovacuum processes if necessary to - prevent transaction ID wraparound. See for more information. + will launch anti-wraparound autovacuums. See for more information. @@ -8536,20 +8537,17 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; Specifies the maximum age (in transactions) that a table's pg_class.relfrozenxid field can - attain before a VACUUM operation is forced - to prevent transaction ID wraparound within the table. - Note that the system will launch autovacuum processes to - prevent wraparound even when autovacuum is otherwise disabled. + attain before an anti-wraparound autovacuum is forced for the table. + Note that the system will launch anti-wraparound autovacuum + processes even when autovacuum is otherwise disabled. - Vacuum also allows removal of old files from the - pg_xact subdirectory, which is why the default - is a relatively low 200 million transactions. + The default is 200 million transactions. This parameter can only be set at server start, but the setting can be reduced for individual tables by changing table storage parameters. - For more information see . + For more information see . @@ -8565,20 +8563,16 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; Specifies the maximum age (in multixacts) that a table's pg_class.relminmxid field can - attain before a VACUUM operation is forced to - prevent multixact ID wraparound within the table. - Note that the system will launch autovacuum processes to - prevent wraparound even when autovacuum is otherwise disabled. + attain before an anti-wraparound autovacuum is forced for the table. + Note that the system will launch anti-wraparound autovacuum + processes even when autovacuum is otherwise disabled. - Vacuuming multixacts also allows removal of old files from the - pg_multixact/members and pg_multixact/offsets - subdirectories, which is why the default is a relatively low - 400 million multixacts. + The default is 400 million Multixact IDs. This parameter can only be set at server start, but the setting can be reduced for individual tables by changing table storage parameters. - For more information see . + For more information see . @@ -9282,10 +9276,11 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; set this value anywhere from zero to two billion, VACUUM will silently limit the effective value to 95% of , so that a - periodic manual VACUUM has a chance to run before an - anti-wraparound autovacuum is launched for the table. For more - information see - . + standard autovacuum (or a manual VACUUM) has a + chance to run using VACUUM's aggressive strategy + before an anti-wraparound autovacuum is launched for the table. For + more information see and . @@ -9307,7 +9302,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; the value of , so that there is not an unreasonably short time between forced autovacuums. For more information see . + linkend="vacuum-freezing-xid-space"/>. @@ -9324,9 +9319,11 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; pg_class.relfrozenxid field can attain before VACUUM takes extraordinary measures to avoid system-wide transaction ID - wraparound failure. This is VACUUM's - strategy of last resort. The failsafe typically triggers - when an autovacuum to prevent transaction ID wraparound has + exhaustion (see ). + This is VACUUM's strategy of last resort. The + failsafe typically triggers when an + anti-wraparound + autovacuum has already been running for some time, though it's possible for the failsafe to trigger during any VACUUM. @@ -9344,7 +9341,8 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; set this value anywhere from zero to 2.1 billion, VACUUM will silently adjust the effective value to no less than 105% of . + linkend="guc-autovacuum-freeze-max-age"/>. For more + information see . @@ -9366,9 +9364,11 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; Although users can set this value anywhere from zero to two billion, VACUUM will silently limit the effective value to 95% of , so that a - periodic manual VACUUM has a chance to run before an - anti-wraparound is launched for the table. - For more information see . + standard autovacuum (or a manual VACUUM) has a + chance to run using VACUUM's aggressive strategy + before an anti-wraparound autovacuum is launched for the table. For + more information see and . @@ -9389,7 +9389,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; the value of , so that there is not an unreasonably short time between forced autovacuums. - For more information see . + For more information see . @@ -9406,9 +9406,11 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; pg_class.relminmxid field can attain before VACUUM takes extraordinary measures to avoid system-wide multixact ID - wraparound failure. This is VACUUM's - strategy of last resort. The failsafe typically triggers when - an autovacuum to prevent transaction ID wraparound has already + exhaustion (see ). + This is VACUUM's strategy of last resort. The + failsafe typically triggers when an + anti-wraparound + autovacuum has already been running for some time, though it's possible for the failsafe to trigger during any VACUUM. @@ -9422,7 +9424,8 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; this value anywhere from zero to 2.1 billion, VACUUM will silently adjust the effective value to no less than 105% of . + linkend="guc-autovacuum-multixact-freeze-max-age"/>. For more + information see . diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index cbd3aa804..cc6499e36 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -352,8 +352,8 @@ postgres=# select * from pg_logical_slot_get_changes('regression_slot', NULL, NU even when there is no connection using them. This consumes storage because neither required WAL nor required rows from the system catalogs can be removed by VACUUM as long as they are required by a replication - slot. In extreme cases this could cause the database to shut down to prevent - transaction ID wraparound (see ). + slot. In extreme cases this could cause the database to refuse to allocate new + transaction IDs (see ). So if a slot is no longer required it should be dropped. diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml index f00442564..abdf01009 100644 --- a/doc/src/sgml/maintenance.sgml +++ b/doc/src/sgml/maintenance.sgml @@ -148,13 +148,8 @@ vacuum insert threshold = vacuum base insert threshold + vacuum insert scale fac . Such vacuums may allow portions of the table to be marked as all visible and also allow tuples to be frozen, which - can reduce the work required in subsequent vacuums. - For tables which receive INSERT operations but no or - almost no UPDATE/DELETE operations, - it may be beneficial to lower the table's - as this may allow - tuples to be frozen by earlier vacuums. The number of obsolete tuples and - the number of inserted tuples are obtained from the cumulative statistics system; + can reduce the work required in subsequent vacuums. The number of obsolete tuples + and the number of inserted tuples are obtained from the cumulative statistics system; it is a semi-accurate count updated by each UPDATE, DELETE and INSERT operation. (It is only semi-accurate because some information might be lost under heavy @@ -211,10 +206,11 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu SHARE UPDATE EXCLUSIVE lock held by autovacuum, lock acquisition will interrupt the autovacuum. For conflicting lock modes, see . However, if the autovacuum - is running to prevent transaction ID wraparound (i.e., the autovacuum query - name in the pg_stat_activity view ends with + is an anti-wraparound autovacuum (i.e., the autovacuum query name in the + pg_stat_activity view ends with (to prevent wraparound)), the autovacuum is not - automatically interrupted. + automatically interrupted. See + for more details on anti-wraparound autovacuums. @@ -272,15 +268,21 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu - To protect against loss of very old data due to - transaction ID wraparound or - multixact ID wraparound. + To maintain the system's ability to allocate transaction IDs + through freezing. To update the visibility map, which speeds up index-only - scans. + scans, and helps the next VACUUM + operation avoid needlessly scanning already-frozen pages. + + + + To enable truncation of obsolescent transaction status + information in structures such as pg_xact for the + entire cluster. @@ -477,303 +479,756 @@ analyze threshold = analyze base threshold + analyze scale factor * number of tu - - Preventing Transaction ID Wraparound Failures - - - transaction ID - wraparound - + + Freezing to manage the transaction ID space - wraparound - of transaction IDs + Freezing + of transaction IDs and Multixact IDs - PostgreSQL's MVCC transaction semantics depend on - being able to compare transaction - ID numbers (XID) to determine - whether or not the row is visible to each query's MVCC snapshot - (see ). But since - on-disk storage of transaction IDs in heap pages uses a truncated - 32-bit representation to save space (rather than the full 64-bit - representation), it is necessary to vacuum every table in every - database at least once every two billion - transactions (though far more frequent vacuuming is typical). + VACUUM often marks some of the pages that it scans + frozen, indicating that all eligible rows on the page + were inserted by a transaction that committed sufficiently far in the past + that the effects of the inserting transaction are certain to be visible to + all current and future transactions. The specific transaction ID number + (XID) stored in a frozen heap row's + xmin field is no longer needed to determine its + visibility. Furthermore, when a row undergoing freezing has an XID set in + its xmax field (e.g., an XID left behind by an + earlier SELECT FOR UPDATE row locker), the + xmax field's XID is usually also removed. - - controls how old an XID value has to be before rows bearing that XID will be - frozen. Increasing this setting may avoid unnecessary work if the - rows that would otherwise be frozen will soon be modified again, - but decreasing this setting increases - the number of transactions that can elapse before the table must be - vacuumed again. + Once frozen, heap pages are self-contained. Every query + can read all of the page's rows in a way that assumes that the inserting + transaction committed and is visible to its MVCC + snapshot. No query will ever have to consult external transaction status + metadata to interpret the page's contents, either. In particular, + pg_xact transaction XID commit/abort status lookups + won't occur during query execution. - VACUUM uses the visibility map - to determine which pages of a table must be scanned. Normally, it - will skip pages that don't have any dead row versions even if those pages - might still have row versions with old XID values. Therefore, normal - VACUUMs won't always freeze every old row version in the table. - When that happens, VACUUM will eventually need to perform an - aggressive vacuum, which will freeze all eligible unfrozen - XID and MXID values, including those from all-visible but not all-frozen pages. - In practice most tables require periodic aggressive vacuuming. - - controls when VACUUM does that: all-visible but not all-frozen - pages are scanned if the number of transactions that have passed since the - last such scan is greater than vacuum_freeze_table_age minus - vacuum_freeze_min_age. Setting - vacuum_freeze_table_age to 0 forces VACUUM to - always use its aggressive strategy. + Freezing is a WAL-logged operation, so when + VACUUM freezes a heap page, any copy of the page + located on a physical replication standby server will itself be + frozen shortly thereafter (when the relevant + FREEZE_PAGE WAL record is replayed + on the standby). Queries that run on physical replication standbys avoid + pg_xact lookups when reading from frozen pages, just + like queries that run on the primary server + + + In this regard, freezing is unlike setting transaction status + hint bits in tuple headers: setting hint bits doesn't + usually need to be WAL-logged, and can take place on + physical replication standby servers without input from the primary + server. Hint bits exist to allow query execution to avoid repeated + pg_xact lookups for the same tuples, strictly as an + optimization. On the other hand, freezing exists because the system + needs to reliably remove pg_xact dependencies from + individual tuples. + + . - The maximum time that a table can go unvacuumed is two billion - transactions minus the vacuum_freeze_min_age value at - the time of the last aggressive vacuum. If it were to go - unvacuumed for longer than - that, data loss could result. To ensure that this does not happen, - autovacuum is invoked on any table that might contain unfrozen rows with - XIDs older than the age specified by the configuration parameter . (This will happen even if - autovacuum is disabled.) + VACUUM generally postpones some freezing work as an + optimization, but VACUUM cannot delay freezing forever. + Since on-disk storage of transaction IDs in heap row headers uses a + truncated partial 32-bit representation to save space (rather than the + full 64-bit representation used in other contexts), it plays a crucial + role in enabling management of the XID + address space by VACUUM. If, for whatever + reason, VACUUM is unable to freeze older XIDs on behalf + of an application that continues to require XID allocations, the system + will eventually refuse to allocate + transaction IDs due to transaction ID exhaustion (though this is + unlikely to occur unless autovacuum is configured incorrectly). - This implies that if a table is not otherwise vacuumed, - autovacuum will be invoked on it approximately once every - autovacuum_freeze_max_age minus - vacuum_freeze_min_age transactions. - For tables that are regularly vacuumed for space reclamation purposes, - this is of little importance. However, for static tables - (including tables that receive inserts, but no updates or deletes), - there is no need to vacuum for space reclamation, so it can - be useful to try to maximize the interval between forced autovacuums - on very large static tables. Obviously one can do this either by - increasing autovacuum_freeze_max_age or decreasing - vacuum_freeze_min_age. + controls when freezing takes + place. When VACUUM scans a heap page containing even + one XID that has already attained an age exceeding this value, the page is + frozen. + + + + Multixact ID + Freezing of + + + + Multixact IDs support row locking by multiple + transactions. Since there is only limited space in a heap tuple header to store lock + information, that information is encoded as a multiple transaction + ID, or Multixact ID for short, whenever there is more than one + transaction concurrently locking a row. Information about which + transaction IDs are included in any particular Multixact ID is stored + separately in pg_multixact. Only the Multixact ID + itself (a 32-bit integer) appears in the tuple's + xmax field. This creates a dependency on + external Multixact ID transaction status information. This is similar to + the dependency ordinary unfrozen XIDs have on commit status information + from pg_xact. VACUUM must + therefore occasionally remove Multixact IDs from tuples during freezing. - The effective maximum for vacuum_freeze_table_age is 0.95 * - autovacuum_freeze_max_age; a setting higher than that will be - capped to the maximum. A value higher than - autovacuum_freeze_max_age wouldn't make sense because an - anti-wraparound autovacuum would be triggered at that point anyway, and - the 0.95 multiplier leaves some breathing room to run a manual - VACUUM before that happens. As a rule of thumb, - vacuum_freeze_table_age should be set to a value somewhat - below autovacuum_freeze_max_age, leaving enough gap so that - a regularly scheduled VACUUM or an autovacuum triggered by - normal delete and update activity is run in that window. Setting it too - close could lead to anti-wraparound autovacuums, even though the table - was recently vacuumed to reclaim space, whereas lower values lead to more - frequent aggressive vacuuming. + also controls when + freezing takes place. It is analogous to + vacuum_freeze_min_age, but age is + expressed in Multixact ID units. Lowering + vacuum_multixact_freeze_min_age + forces VACUUM to process + xmax fields with a Multixact ID in cases where + it would otherwise postpone the work of processing + xmax until the next VACUUM + + + Freezing of xmax fields + (whether they contain an XID or a Multixact ID) generally means clearing + xmax from a tuple header. + VACUUM may occasionally encounter an individual + Multixact ID that must be removed to advance the table's + relminmxid by the required amount, which can + only be processed by generating a replacement Multixact ID (containing + just the non-removable subset of member XIDs from the original Multixact + ID), and then setting xmax to the + new/replacement Multixact ID value. + + . The setting generally doesn't significantly influence the + total number of pages VACUUM freezes, even in tables + containing many Multixact IDs. This is because VACUUM + generally prefers proactive processing for most individual + xmax fields that contain a Multixact ID (eager + proactive processing is typically cheaper). - The sole disadvantage of increasing autovacuum_freeze_max_age - (and vacuum_freeze_table_age along with it) is that - the pg_xact and pg_commit_ts - subdirectories of the database cluster will take more space, because it - must store the commit status and (if track_commit_timestamp is - enabled) timestamp of all transactions back to - the autovacuum_freeze_max_age horizon. The commit status uses - two bits per transaction, so if - autovacuum_freeze_max_age is set to its maximum allowed value - of two billion, pg_xact can be expected to grow to about half - a gigabyte and pg_commit_ts to about 20GB. If this - is trivial compared to your total database size, - setting autovacuum_freeze_max_age to its maximum allowed value - is recommended. Otherwise, set it depending on what you are willing to - allow for pg_xact and pg_commit_ts storage. - (The default, 200 million transactions, translates to about 50MB - of pg_xact storage and about 2GB of pg_commit_ts - storage.) + Managing the added WAL volume from freezing over time + is a vital consideration for VACUUM. It is why + VACUUM doesn't just freeze every eligible tuple at the + earliest opportunity: the WAL written to freeze a + page's tuples is wasted in cases where the resulting frozen tuples are + soon deleted or updated anyway. It's also why VACUUM + will freeze all eligible tuples from a heap page once + the decision to freeze at least one tuple is taken: at that point, the + added cost of freezing all eligible tuples eagerly (measured in + extra bytes of WAL written) is far lower + than the probable cost of deferring freezing until a future + VACUUM operation against the same table. Furthermore, + once the page is frozen, it can generally be marked as all-frozen within the + visibility map immediately afterwards. - - One disadvantage of decreasing vacuum_freeze_min_age is that - it might cause VACUUM to do useless work: freezing a row - version is a waste of time if the row is modified - soon thereafter (causing it to acquire a new XID). So the setting should - be large enough that rows are not frozen until they are unlikely to change - any more. - + + + In PostgreSQL versions before 16, + VACUUM triggered freezing at the level of individual + xmin and xmax + fields. Freezing only affected the exact XIDs that had already attained + an age of vacuum_freeze_min_age or greater. + + - To track the age of the oldest unfrozen XIDs in a database, - VACUUM stores XID - statistics in the system tables pg_class and - pg_database. In particular, - the relfrozenxid column of a table's - pg_class row contains the oldest remaining unfrozen - XID at the end of the most recent VACUUM that successfully - advanced relfrozenxid (typically the most recent - aggressive VACUUM). Similarly, the - datfrozenxid column of a database's - pg_database row is a lower bound on the unfrozen XIDs - appearing in that database — it is just the minimum of the - per-table relfrozenxid values within the database. - A convenient way to - examine this information is to execute queries such as: - - -SELECT c.oid::regclass as table_name, - greatest(age(c.relfrozenxid),age(t.relfrozenxid)) as age -FROM pg_class c -LEFT JOIN pg_class t ON c.reltoastrelid = t.oid -WHERE c.relkind IN ('r', 'm'); - -SELECT datname, age(datfrozenxid) FROM pg_database; - - - The age column measures the number of transactions from the - cutoff XID to the current transaction's XID. + VACUUM also triggers the freezing of a page in cases + where it already proved necessary to write out a full page image + (FPI) as part of a WAL record + describing how dead tuples were removed + + Actually, the freeze on an FPI write + mechanism isn't just used when VACUUM needs to + generate an FPI (as torn page protection) for + inclusion in a WAL record describing how dead tuples + were removed. The FPI mechanism also triggers when + hint bits are set by VACUUM, if and only if it + necessitates writing an FPI. The need to write a + WAL record to set hint bits only arises when + is enabled in + postgresql.conf, or when data checksums were + enabled when the cluster was initialized with . + + (see for background + information about how FPIs provide torn page + protection). This freeze on an FPI + write batching mechanism avoids an expected additional + FPI for the same page later on (this is the probable + outcome of lazily deferring freezing until vacuum_freeze_min_age + forces it). In effect, VACUUM generates slightly more + WAL in the short term with the aim of ultimately + needing to generate much less WAL in the long term. - When the VACUUM command's VERBOSE - parameter is specified, VACUUM prints various - statistics about the table. This includes information about how - relfrozenxid and - relminmxid advanced, and the number of - newly frozen pages. The same details appear in the server log when - autovacuum logging (controlled by ) reports on a - VACUUM operation executed by autovacuum. + For tables that receive INSERT operations, but few or + no UPDATE/DELETE operations, it + might be beneficial to lower + for the table. This makes VACUUM freeze the table's + pages eagerly during earlier autovacuums triggered by + , which + improves performance stability for some workloads. - - VACUUM normally only scans pages that have been modified - since the last vacuum, but relfrozenxid can only be - advanced when every page of the table - that might contain unfrozen XIDs is scanned. This happens when - relfrozenxid is more than - vacuum_freeze_table_age transactions old, when - VACUUM's FREEZE option is used, or when all - pages that are not already all-frozen happen to - require vacuuming to remove dead row versions. When VACUUM - scans every page in the table that is not already all-frozen, it should - set age(relfrozenxid) to a value just a little more than the - vacuum_freeze_min_age setting - that was used (more by the number of transactions started since the - VACUUM started). VACUUM - will set relfrozenxid to the oldest XID - that remains in the table, so it's possible that the final value - will be much more recent than strictly required. - If no relfrozenxid-advancing - VACUUM is issued on the table until - autovacuum_freeze_max_age is reached, an autovacuum will soon - be forced for the table. - + + Aggressive <command>VACUUM</command> - - If for some reason autovacuum fails to clear old XIDs from a table, the - system will begin to emit warning messages like this when the database's - oldest XIDs reach forty million transactions from the wraparound point: + + transaction ID + wraparound + + + + wraparound + of transaction IDs and Multixact IDs + + + + As noted already, freezing doesn't just allow queries to avoid lookups of + subsidiary transaction status information in structures such as + pg_xact. Freezing also plays a crucial role in + enabling transaction ID address space management by + VACUUM (and autovacuum). VACUUM + maintains information about the oldest unfrozen XID that remains in the + table when it uses its aggressive strategy. + + + + Aggressive VACUUM updates the table's pg_class.relfrozenxid + to whatever XID was the oldest observed XID that + VACUUM didn't freeze still + remaining at the end of processing. The table's + relfrozenxid advances by a + certain number of XIDs (relative to the previous value set during the last + aggressive VACUUM) as progress on freezing the oldest + pages in the table permits. Aggressive VACUUM will + occasionally need to advance the whole database's pg_database.datfrozenxid + afterwards, too — this is the minimum of the per-table + relfrozenxid values (i.e., the earliest + relfrozenxid) within the database. + + + + Aggressive VACUUM may need to perform significant + amounts of catch-up freezing missed by earlier + non-aggressive VACUUMs, because non-aggressive + VACUUM sometimes allows unfrozen pages to build up. + + + + Over time, aggressive autovacuuming has two beneficial effects on the + system as a whole: + + + + It keeps track of the oldest remaining unfrozen transaction ID + in the entire database + cluster (i.e., the oldest transaction ID across every + table in every database). + + + It avoids a cluster-wide oldest unfrozen transaction ID that + is too old. + + + + + + The maximum XID age that the system can tolerate (i.e., the maximum + distance between the oldest unfrozen transaction ID in any + table in the database cluster and the next unallocated transaction ID) is + about 2.1 billion transaction IDs. This maximum XID age + invariant makes it fundamentally impossible to postpone aggressive + VACUUMs (and freezing) forever. While there is no + simple formula for determining an oldest XID age for + database administrators to target, the invariant imposes a 2.1 billion + XID age hard limit — so there is a clear point + at which unfrozen XIDs should always be considered + too old, regardless of individual application requirements + or workload characteristics. If the hard limit is reached, the system + experiences transaction ID + exhaustion, which temporarily prevents the allocation of new + permanent transaction IDs. The system will only regain the ability to + allocate new transaction IDs when VACUUM succeeds in + advancing the oldest datfrozenxid in the cluster + (following an aggressive VACUUM that runs to + completion against the table with the oldest + relfrozenxid). + + + + Aggressive VACUUM also maintains the pg_class.relminmxid and + pg_database.datminmxid + fields. These are needed to track the oldest Multixact ID in the table + and database, respectively. There are analogous rules, driven by + analogous considerations about managing the Multixact ID space. This + doesn't usually affect aggressive vacuuming requirements to a noticeable + degree, but can in databases that consume more Multixact IDs than + transaction IDs. + + + + + VACUUM may not always freeze tuple + xmin XIDs that have reached + vacuum_freeze_min_age in age. The basic eligibility + criteria for freezing is the same as the criteria that determines if a + deleted tuple is safe for VACUUM to remove: the + XID-based removable cutoff (this is one of the + details that appears in the server log's reports on autovacuum + + + Autovacuum's log reports appear in the server log for autovacuums + whose VACUUM takes longer than a threshold + controlled by . + Manual VACUUMs output the same details as + INFO messages when the VACUUM + command's VERBOSE option is used (note that manual + VACUUMs never generate reports in the server log). + + ). + + + In extreme cases, a long-running transaction can hold back every + VACUUM's removable cutoff for so + long that the system experiences + transaction ID exhaustion. + See for details on how to monitor + relfrozenxid and + relminmxid age to avoid + transaction ID/Multixact ID exhaustion. + + + These issues can be debugged by following autovacuum log reports from + the server log over time: the log reports will include information about + the age of each VACUUM's removable + cutoff at the point the VACUUM ended. + It may be useful to correlate the use of a cutoff with an excessively + high age with application-level problems such as long-running + transactions. + + + + + The 2.1 billion XIDs maximum XID age invariant must be + preserved because transaction IDs stored in heap tuple headers use a truncated + 32-bit representation (rather than the full 64-bit representation used in + other contexts). Since all unfrozen transaction IDs from heap tuple + headers must be from the same transaction ID epoch + (or from a space in the 64-bit representation that spans two adjoining + transaction ID epochs), there isn't any need to include a separate epoch + field in each tuple header (see + for further details). This scheme requires much less on-disk storage + space than a design that stores full 64-bit XIDs (consisting of a 32-bit + epoch and a 32-bit partial XID) in heap tuple headers. On the other + hand, it constrains the system's ability to allocate new XIDs in the + worst case scenario where transaction ID exhaustion occurs. + + + + There is only one major behavioral difference + between aggressive VACUUM and non-aggressive + VACUUM: non-aggressive VACUUM skips + pages marked as all-visible using the visibility map, whereas aggressive + VACUUM only skips the subset of pages that are both + all-visible and all-frozen. In other words, pages + that are just all-visible at the beginning of an + aggressive VACUUM must be scanned, not skipped. + Scanning existing all-visible pages is necessary to determine the oldest + unfrozen XID that will remain in the table at the end of an aggressive + VACUUM. + + + + + In practice, most tables require periodic aggressive vacuuming. + However, some individual non-aggressive VACUUM + operations can advance the table's + relfrozenxid and/or + relminmxid. + + + This happens whenever a non-aggressive VACUUM notices + that it is safe without incurring any added cost from scanning + extra pages. It is most common in small, frequently + modified tables. + + + + + Non-aggressive VACUUMs can sometimes overlook older + XIDs from existing all-visible pages (due to their policy of always + skipping all-visible pages). Over time, this can even lead to a + significant build-up of unfrozen pages in one table (accumulated + all-visible pages that remain unfrozen). When that happens, it is + inevitable that an aggressive VACUUM will eventually + need to perform catch-up freezing that clears the table's + backlog of unfrozen pages. + + + + There is also one minor behavioral difference + between aggressive VACUUM and non-aggressive + VACUUM: only aggressive VACUUM is + required to sometimes wait for a page-level cleanup lock when a page is + scanned and observed to contain transaction IDs/Multixact IDs that + must be frozen. This difference exists because + aggressive VACUUM is strictly required to advance + relfrozenxid and/or + relminmxid to + sufficiently recent values + + + Aggressive VACUUM is (somewhat arbitrarily) required + to freeze all pages containing transaction IDs older than + vacuum_freeze_min_age and/or Multixact ID values + older than vacuum_multixact_freeze_min_age, at a + minimum. + + . The behavior can lead to occasional waits for a conflicting + buffer pin to be released by another backend. These waits are + imperceptible and harmless most of the time. In extreme cases there can + be extended waits, which can be observed under the + BufferPin wait event in the + pg_stat_activity view. See . + + + + + Catch-up freezing is not caused by any difference in how + vacuum_freeze_min_age is applied by each type of + VACUUM. It is an indirect result of + vacuum_freeze_min_age only being applied to those + pages that VACUUM scans (and cleanup locks) in the + first place. Therefore, it can be difficult to tune + vacuum_freeze_min_age, especially for tables that + receive frequent non-aggressive VACUUMs and + infrequent aggressive VACUUMs. + + + + + + Autovacuum server log reports show how many transaction IDs + relfrozenxid advanced by (if at all), and how + many Multixact IDs relminmxid advanced by (if + at all). + + + The number of pages frozen, and the number of pages scanned (i.e., the + number of pages processed because they could not + skipped using the visibility map) are also shown. This can provide + useful guidance when tuning freezing-related settings, particularly + vacuum_freeze_table_age and + vacuum_freeze_min_age. + + + + + controls when + VACUUM uses its aggressive strategy. If + age(relfrozenxid) exceeds + vacuum_freeze_table_age at the start of + VACUUM, VACUUM will employ its + aggressive strategy; otherwise, its standard non-aggressive strategy is + employed. Setting vacuum_freeze_table_age to 0 forces + VACUUM to always use its aggressive strategy. + + + + also controls + when VACUUM uses its aggressive strategy. This is an + independent Multixact ID based trigger for aggressive + VACUUM, which works just like + vacuum_freeze_table_age. It is applied against + mxid_age(relminmxid) at the start of each + VACUUM. + + + + It doesn't matter if it was vacuum_freeze_table_age or + vacuum_multixact_freeze_table_age that triggered + VACUUM's decision to use its aggressive strategy. + Every aggressive VACUUM will + advance relfrozenxid and + relminmxid by applying the same generic policy + that controls which pages are frozen. + + + + The default vacuum_freeze_table_age and + vacuum_multixact_freeze_table_age settings are + relatively low values. The vacuum_freeze_table_age + and vacuum_freeze_min_age defaults are intended to + limit the system to using only about 10% of the available transaction ID + space at any one time. This leaves the system with a generous amount of + slack capacity that allows XID allocations to continue in + the event of unforeseen problems with autovacuum and/or the application. + There might only be a negligible benefit from higher settings that aim to + reduce the number of VACUUMs that use the aggressive + strategy, in any case. Some applications may even + benefit from tuning that makes autovacuum perform + aggressive VACUUMs more often. If individual + aggressive VACUUMs can perform significantly less + catch-up freezing as a result, overall transaction + processing throughput is likely to be more stable and predictable. + + + + + Anti-Wraparound Autovacuums + + + To ensure that every table has its + relfrozenxid (and + relminmxid) advanced at regular intervals, + even in the case of completely static tables, autovacuum runs against any + table when it attains an age considered too far in the past. These are + anti-wraparound autovacuums. In practice, all + anti-wraparound autovacuums will use VACUUM's + aggressive strategy (if they didn't, it would defeat the whole purpose of + anti-wraparound autovacuuming). + + + + controls when the + autovacuum daemon launches anti-wraparound autovacuums. If the + age(relfrozenxid) of a table exceeds + autovacuum_freeze_max_age when the autovacuum daemon + periodically examines the database (which happens once every seconds), then an anti-wraparound + autovacuum is launched against the table. + + + + also controls + when the autovacuum daemon launches anti-wraparound autovacuums. It is + an independent Multixact ID based trigger for anti-wraparound + autovacuuming. If the mxid_age(relminmxid) of a table + exceeds autovacuum_multixact_freeze_max_age when the + autovacuum daemon periodically examines the database + + + Autovacuum may use a lower effective Multixact ID age + than the autovacuum_multixact_freeze_max_age setting + in postgresql.conf, though. Applying a lower + effective value like this avoids allowing the + pg_multixact/members SLRU + storage area to continue to grow in size for long, once its size + reaches 2GB. + See . + + , then an anti-wraparound autovacuum is launched against the + table. + + + + Use of VACUUM's aggressive strategy during + anti-wraparound autovacuuming is certain, because the effective value of + vacuum_freeze_table_age is silently limited to an + effective value no greater than 95% of the current value of + autovacuum_freeze_max_age. Similarly, the effective + value of vacuum_multixact_freeze_table_age is silently + limited to a value no greater than 95% of the current value of + autovacuum_multixact_freeze_max_age. + + + + It doesn't matter if it was autovacuum_freeze_max_age + or autovacuum_multixact_freeze_max_age that triggered + an anti-wraparound autovacuum. Every + anti-wraparound autovacuum will be an aggressive + VACUUM, and will therefore advance + relfrozenxid and + relminmxid by applying the same generic policy + that controls which pages are frozen. + + + + Anti-wraparound autovacuums are intended for static (and mostly static) + tables. There is no reason to expect that a table receiving continual + row inserts and/or row modifications will ever require an anti-wraparound + autovacuum. As a rule of thumb, + autovacuum_freeze_max_age should be set to a value + somewhat above vacuum_freeze_table_age, so that there + is a long window during which any autovacuum triggered by inserts, + updates, or deletes (or any manually issued VACUUM) + will become an aggressive VACUUM. This has the + advantage of allowing aggressive vacuuming to take place at a time when + vacuuming was required anyway. Each aggressive VACUUM + can therefore be expected to perform just as much useful work on + recovering disk space as an equivalent non-aggressive + VACUUM would have (had the non-aggressive strategy + been chosen instead). + + + + Aggressive/anti-wraparound differences + + Aggressive VACUUM is a special type of + VACUUM. It must advance + relfrozenxid up to a value that was no + greater than vacuum_freeze_min_age in age as of the + start of the VACUUM operation. + + + Anti-wraparound autovacuum is a special type of autovacuum. Its purpose + is to ensure that relfrozenxid advances when + no earlier VACUUM could advance it in passing — + often because no VACUUM has run against the table for + an extended period. + + + There is only one runtime behavioral difference between anti-wraparound + autovacuums and other autovacuums that run aggressive + VACUUMs: anti-wraparound autovacuums cannot + be autocancelled. This means that autovacuum workers that + perform anti-wraparound autovacuuming do not yield to conflicting + relation-level lock requests (e.g., from ALTER + TABLE). See for + a full explanation. + + + + + In practice, anti-wraparound autovacuum is very likely to be the type of + autovacuum that updates the oldest relfrozenxid + in each database to a more recent value due to the presence of completely + static tables + + Anti-wraparound autovacuum is all but guaranteed to advance the oldest + relfrozenxid/relminmxid + (and therefore to advance + datfrozenxid/datminmxid) + in practice because in practice there is all but guaranteed to be at + least one totally static table that never gets an aggressive + VACUUM for any other reason (often just a tiny, + completely static system catalog table). + + . As discussed in , + datfrozenxid only advances when the oldest + relfrozenxid in the database advances + (relminmxid likewise only advances when the + earliest datminmxid in the database advances). + This implies that anti-wraparound autovacuum is also + very likely to be involved when any database's + datfrozenxid/datminmxid + advances (and when the cluster-wide earliest unfrozen transaction + ID/Multixact ID is advanced to a more recent value, in turn). + + + + It follows that autovacuum_freeze_max_age is usually + the limiting factor for advancing the cluster-wide oldest unfrozen + transaction ID found in pg_control + (the cluster-wide oldest unfrozen Multixact ID might occasionally be + influenced by autovacuum_multixact_freeze_max_age, + too). This usually isn't much of a concern in itself, since it generally + doesn't predict anything about how far behind autovacuum is with freezing + physical heap pages. Note, however, that this effect + can significantly impact the amount of space + required to store transaction status information. The oldest transaction + status information (which is stored in external structures such as + pg_xact) cannot safely be truncated until + VACUUM can ascertain that there are no references to + the oldest entries remaining in any table, from any database. See for further details. + + + + + Transaction ID exhaustion + + If for some reason autovacuum fails to advance any table's + relfrozenxid for an extended period (during + which transaction IDs continue to be allocated), the system will begin to + emit warning messages once the database's oldest XIDs attain an age + within forty million transactions of the 2.1 billion XID hard limit + described in . For example: WARNING: database "mydb" must be vacuumed within 39985967 transactions HINT: To avoid a database shutdown, execute a database-wide VACUUM in that database. - (A manual VACUUM should fix the problem, as suggested by the - hint; but note that the VACUUM must be performed by a - superuser, else it will fail to process system catalogs and thus not - be able to advance the database's datfrozenxid.) - If these warnings are - ignored, the system will shut down and refuse to start any new - transactions once there are fewer than three million transactions left - until wraparound: + (A manual VACUUM should fix the problem, as suggested by the + hint; but note that the VACUUM must be performed by a + superuser, else it will fail to process system catalogs and thus not + be able to advance the database's datfrozenxid.) + If these warnings are ignored, the system will eventually refuse + to allocate new transaction IDs. This happens at the point that + there are fewer than three million transactions left: ERROR: database is not accepting commands to avoid wraparound data loss in database "mydb" HINT: Stop the postmaster and vacuum that database in single-user mode. - The three-million-transaction safety margin exists to let the - administrator recover without data loss, by manually executing the - required VACUUM commands. However, since the system will not - execute commands once it has gone into the safety shutdown mode, - the only way to do this is to stop the server and start the server in single-user - mode to execute VACUUM. The shutdown mode is not enforced - in single-user mode. See the reference - page for details about using single-user mode. - - - - Multixacts and Wraparound - - - MultiXactId - - - - wraparound - of multixact IDs - - - - Multixact IDs are used to support row locking by - multiple transactions. Since there is only limited space in a tuple - header to store lock information, that information is encoded as - a multiple transaction ID, or multixact ID for short, - whenever there is more than one transaction concurrently locking a - row. Information about which transaction IDs are included in any - particular multixact ID is stored separately in - the pg_multixact subdirectory, and only the multixact ID - appears in the xmax field in the tuple header. - Like transaction IDs, multixact IDs are implemented as a - 32-bit counter and corresponding storage, all of which requires - careful aging management, storage cleanup, and wraparound handling. - There is a separate storage area which holds the list of members in - each multixact, which also uses a 32-bit counter and which must also - be managed. + The three-million-transaction safety margin exists to let the + administrator recover without data loss, by manually executing the + required VACUUM commands. However, since the system will not + execute commands once it has gone into the safety shutdown mode, + the only way to do this is to stop the server and start the server in single-user + mode to execute VACUUM. The shutdown mode is not enforced + in single-user mode. See the reference + page for details about using single-user mode. - Whenever VACUUM scans any part of a table, it will replace - any multixact ID it encounters which is older than - - by a different value, which can be the zero value, a single - transaction ID, or a newer multixact ID. For each table, - pg_class.relminmxid stores the oldest - possible multixact ID still appearing in any tuple of that table. - If this value is older than - , an aggressive - vacuum is forced. As discussed in the previous section, an aggressive - vacuum means that only those pages which are known to be all-frozen will - be skipped. mxid_age() can be used on - pg_class.relminmxid to find its age. + A similar safety mechanism is used to prevent Multixact ID allocations + when any table's relminmxid is dangerously far + in the past: Multixact ID exhaustion. If the system isn't experiencing + transaction ID exhaustion, Multixact ID exhaustion can be fixed + non-invasively by running a manual VACUUM without + entering single-user mode (otherwise follow the procedure for transaction + ID exhaustion). See for details + on how to determine which table's relminmxid + is dangerously far in the past. - - Aggressive VACUUMs, regardless of what causes - them, are guaranteed to be able to advance - the table's relminmxid. - Eventually, as all tables in all databases are scanned and their - oldest multixact values are advanced, on-disk storage for older - multixacts can be removed. - - - - As a safety device, an aggressive vacuum scan will - occur for any table whose multixact-age is greater than . Also, if the - storage occupied by multixacts members exceeds 2GB, aggressive vacuum - scans will occur more often for all tables, starting with those that - have the oldest multixact-age. Both of these kinds of aggressive - scans will occur even if autovacuum is nominally disabled. - + + + Autovacuum has two different mechanisms that are designed to avoid + transaction ID exhaustion. The first mechansim is anti-wraparound + autovacuuming. There is an second, independent mechanism, used when + relfrozenxid and/or + relminmxid have already consumed a + significant fraction of the total available transaction ID space: the + failsafe. + + + The failsafe is triggered by VACUUM when the table's + relfrozenxid attains an age of XIDs, or when the table's + relminmxid attains an age of Multixact IDs. This + happens dynamically, when the risk of eventual transaction ID (or + Multixact ID) exhaustion is deemed to outweigh the risks of not + proceeding as planned with ordinary vacuuming. + + + Once the failsafe triggers, VACUUM prioritizes + advancing relfrozenxid and/or + relminmxid to avoid transaction ID + exhaustion. Most notably, VACUUM bypasses any + remaining non-essential maintenance, such as index vacuuming. + + @@ -784,9 +1239,19 @@ HINT: Stop the postmaster and vacuum that database in single-user mode. Vacuum maintains a visibility map for each table to keep track of which pages contain only tuples that are known to be visible to all active transactions (and all future transactions, until the - page is again modified). This has two purposes. First, vacuum - itself can skip such pages on the next run, since there is nothing to - clean up. + page is again modified). A separate bit tracks whether all of the tuples + are frozen. + + + + The visibility map serves two purposes. + + + + First, VACUUM itself can skip such pages on the + next run, since there is nothing to clean up. Even aggressive VACUUMs + can skip pages that are both all-visible and all-frozen. @@ -806,6 +1271,79 @@ HINT: Stop the postmaster and vacuum that database in single-user mode. + + Truncating Transaction Status Information + + + As discussed in , aggressive + autovacuuming plays a critical role in maintaining the XID address space + for the system as a whole. A secondary goal of this whole process is to + enable eventual truncation of the oldest transaction status information in + the database cluster + as a whole. This status information is stored in dedicated simple least-recently-used + (SLRU) caches backed by external storage (see ). Truncation is only possible when + VACUUM can ascertain that there are no references to + the oldest entries remaining in any table, from any database, by taking + the earliest datfrozenxid and + datminmxid among all databases in the cluster. + This isn't a maintenance task that affects individual tables; it's a + maintenance task that affects the whole cluster. + + + + The space required to store transaction status information is likely to be + a low priority for most database administrators. It may occasionally be + useful to limit the maximum storage overhead used for transaction status + information by making anti-wraparound autovacuums happen more frequently. + The frequency of system-wide anti-wraparound autovacuuming increases when + autovacuum_freeze_max_age and + autovacuum_multixact_freeze_max_age are decreased in + postgresql.conf. This approach is effective (at + limiting the storage required for transaction status information) because + the oldest datfrozenxid + and datminmxid in the cluster are very likely + to depend on the frequency of anti-wraparound autovacuuming of completely + static tables. See for + further discussion of the role of anti-wraparound autovacuuming in + advancing the cluster-wide oldest unfrozen transaction ID. + + + + There are two SLRU storage areas associated with + transaction IDs. First, there is pg_xact, which + stores commit/abort status information. Second, there is + pg_commit_ts, which stores transaction commit + timestamps (when is set to + on). The default + autovacuum_freeze_max_age setting of 200 million + transactions translates to about 50MB of pg_xact + storage, and about 2GB of pg_commit_ts storage when + track_commit_timestamp is enabled (it is set to + off by default, which totally avoids the need to store + anything in pg_commit_ts). + + + + There are also two SLRU storage areas associated with + Multixact IDs: pg_multixact/members, and + pg_multixact/offsets. These are logically one + storage area, implemented as two distinct storage areas. There is no + simple formula to determine the storage overhead per Multixact ID, since + Multixact IDs have a variable number of member transaction IDs (this is + what necessitates using two different physical storage areas). Note, + however, that if pg_multixact/members exceeds 2GB, + the effective value of autovacuum_multixact_freeze_max_age + used by autovacuum (and VACUUM) will be lower. This + results in more frequent + anti-wraparound + autovacuums (since that's the only approach that reliably limits + the size of these storage areas). It might also increase the frequency of + aggressive VACUUMs more generally. + + + Updating Planner Statistics diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index 99f7f95c3..0888342ef 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -1032,9 +1032,11 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser BufferPin The server process is waiting for exclusive access to - a data buffer. Buffer pin waits can be protracted if - another process holds an open cursor that last read data from the - buffer in question. See . + a data buffer. Buffer pin waits by aggressive + VACUUM (see ) can + be protracted if another process holds an for details on how to open + cursor that last read data from the buffer in question. + See . @@ -6246,6 +6248,78 @@ FROM pg_stat_get_backend_idset() AS backendid; + + Monitoring table age + + + It is crucial that autovacuum is able to run VACUUM (or + that every table is manually vacuumed) at somewhat regular intervals. Even + completely static tables need to participate in management of the + transaction ID space by VACUUM, as explained in . + + + + This section provides details about how to monitor for this. You might + want to use a third party monitoring and alterting tool for this. Many + have off-the-shelf queries similar to the reference query shown here. + + + + + Temporary tables are not maintained by autovacuum (see ), but nevertheless have the same + requirements for freezing and relfrozenxid + advancement as permanent tables. + + + This means application code that leaks temporary tables can + threaten the availability of the system; eventually, the system will enter + a mode that makes it temporarily unable to allocate new transaction IDs + (see ), since autovacuum's usual + strategies for preventing that from happening cannot be used. + + + + + A convenient way to examine information about + relfrozenxid and + relminmxid is to execute queries such as: + + +SELECT c.oid::regclass as table_name, +greatest(age(c.relfrozenxid), + age(t.relfrozenxid)) as xid_age, +mxid_age(c.relminmxid) +FROM pg_class c +LEFT JOIN pg_class t ON c.reltoastrelid = t.oid +WHERE c.relkind IN ('r', 'm'); + +SELECT datname, +age(datfrozenxid) as xid_age, +mxid_age(datminmxid) +FROM pg_database; + + + The age function returns the number of transactions + from relfrozenxid to the next unallocated + transaction ID. The mxid_age function returns the + number of Multixact IDs from relminmxid to the + next unallocated Multixact ID. + + + + The system should always have significant XID allocation slack capacity. + Ideally, the greatest + age(relfrozenxid)/age(datfrozenxid) + in the system will never be more than a fraction of the 2.1 billion XID + hard limit described in . The default + vacuum_freeze_table_age setting of 200 million + transactions implies that the system should never use significantly more + than about 10% of that hard limit. + + + Viewing Locks diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 10ef699fa..307504acb 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -1514,11 +1514,10 @@ WITH ( MODULUS numeric_literal, REM If true, the autovacuum daemon will perform automatic VACUUM and/or ANALYZE operations on this table following the rules discussed in . - If false, this table will not be autovacuumed, except to prevent - transaction ID wraparound. See for - more about wraparound prevention. - Note that the autovacuum daemon does not run at all (except to prevent - transaction ID wraparound) if the + If false, this table will not be autovacuumed, except to run + anti-wraparound autovacuum (see ). + Note that the autovacuum daemon does not run at all (except to run + anti-wraparound autovacuums) if the parameter is false; setting individual tables' storage parameters does not override that. Therefore there is seldom much point in explicitly setting this storage parameter to true, only diff --git a/doc/src/sgml/ref/prepare_transaction.sgml b/doc/src/sgml/ref/prepare_transaction.sgml index f4f6118ac..719e0b25a 100644 --- a/doc/src/sgml/ref/prepare_transaction.sgml +++ b/doc/src/sgml/ref/prepare_transaction.sgml @@ -126,13 +126,13 @@ PREPARE TRANSACTION transaction_id It is unwise to leave transactions in the prepared state for a long time. This will interfere with the ability of VACUUM to reclaim - storage, and in extreme cases could cause the database to shut down - to prevent transaction ID wraparound (see ). Keep in mind also that the transaction - continues to hold whatever locks it held. The intended usage of the - feature is that a prepared transaction will normally be committed or - rolled back as soon as an external transaction manager has verified that - other databases are also prepared to commit. + storage, and in extreme cases could cause the database to refuse to + allocate new transaction IDs (see ). + Keep in mind also that the transaction continues to hold whatever locks it + held. The intended usage of the feature is that a prepared transaction + will normally be committed or rolled back as soon as an external + transaction manager has verified that other databases are also prepared to + commit. diff --git a/doc/src/sgml/ref/vacuum.sgml b/doc/src/sgml/ref/vacuum.sgml index 57bc4c23e..f8c010123 100644 --- a/doc/src/sgml/ref/vacuum.sgml +++ b/doc/src/sgml/ref/vacuum.sgml @@ -123,7 +123,9 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ aggressive strategy. Specifying FREEZE is equivalent to performing VACUUM with the and @@ -218,11 +220,11 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ ). However, the - wraparound failsafe mechanism controlled by ). However, the failsafe + mechanism controlled by will generally trigger - automatically to avoid transaction ID wraparound failure, and + automatically to avoid transaction ID exhaustion failure, and should be preferred. If index cleanup is not performed regularly, performance may suffer, because as the table is modified indexes will accumulate dead tuples and the table @@ -232,7 +234,7 @@ VACUUM [ FULL ] [ FREEZE ] [ VERBOSE ] [ ANALYZE ] [ mxid_age. - This setting is useful for prioritizing tables to process to prevent - multixact ID wraparound (see - ). + This setting is useful for prioritizing tables to advance + pg_class.relfrozenxid + for via an aggressive VACUUM + (see and + ). For the purposes of this option, the multixact ID age of a relation is @@ -252,9 +254,12 @@ PostgreSQL documentation Only execute the vacuum or analyze commands on tables with a transaction ID age of at least - xid_age. This setting - is useful for prioritizing tables to process to prevent transaction - ID wraparound (see ). + xid_age. This + setting is useful for prioritizing tables to advance + pg_class.relminmxid + for via an aggressive VACUUM + (see and + ). For the purposes of this option, the transaction ID age of a relation diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml index 148fb1b49..6d7ddc685 100644 --- a/doc/src/sgml/storage.sgml +++ b/doc/src/sgml/storage.sgml @@ -664,7 +664,8 @@ This information can also be used by index-only scans to answer queries using only the index tuple. The second bit, if set, means that all tuples on the page have been frozen. -That means that even an anti-wraparound vacuum need not revisit the page. +That means that even an aggressive vacuum (see ) +need not revisit the page. diff --git a/doc/src/sgml/xact.sgml b/doc/src/sgml/xact.sgml index 8a1f9fd6f..3cba050f0 100644 --- a/doc/src/sgml/xact.sgml +++ b/doc/src/sgml/xact.sgml @@ -180,7 +180,7 @@ rows and can be inspected using the extension. Row-level read locks might also require the assignment of multixact IDs (mxid; see ). + linkend="vacuum-freezing-xid-space"/>). -- 2.40.1