Postgres Pro
Downloads
Home   >   mailing lists

Thread: broken tables on hot standby after migration on PostgreSQL 16 (3x times last month)

broken tables on hot standby after migration on PostgreSQL 16 (3x times last month)

From
Pavel Stehule
Date:
Hi

after migration on PostgreSQL 16 I seen 3x times (about every week) broken tables on replica nodes. The query fails with error

ERROR:  could not access status of transaction 1442871302
DETAIL:  Could not open file "pg_xact/0560": No such file or directory

verify_heapam reports

^[[Aprd=# select * from verify_heapam('account_login_history') where blkno = 179036;
 blkno  | offnum | attnum |                                msg                                
--------+--------+--------+-------------------------------------------------------------------
 179036 |     30 |        | xmin 1393743382 precedes oldest valid transaction ID 3:1687012112
 179036 |     31 |        | xmin 1393748413 precedes oldest valid transaction ID 3:1687012112
 179036 |     32 |        | xmin 1393751312 precedes oldest valid transaction ID 3:1687012112
 179036 |     33 |        | xmin 1393763601 precedes oldest valid transaction ID 3:1687012112
 179036 |     34 |        | xmin 1393795606 precedes oldest valid transaction ID 3:1687012112
 179036 |     35 |        | xmin 1393817722 precedes oldest valid transaction ID 3:1687012112
 179036 |     36 |        | xmin 1393821359 precedes oldest valid transaction ID 3:1687012112
 179036 |     37 |        | xmin 1393821373 precedes oldest valid transaction ID 3:1687012112
 179036 |     38 |        | xmin 1393821523 precedes oldest valid transaction ID 3:1687012112
 179036 |     39 |        | xmin 1410429961 precedes oldest valid transaction ID 3:1687012112
 179036 |     40 |        | xmin 1410433593 precedes oldest valid transaction ID 3:1687012112
 179036 |     41 |        | xmin 1410501438 precedes oldest valid transaction ID 3:1687012112
 179036 |     42 |        | xmin 1410511950 precedes oldest valid transaction ID 3:1687012112
 179036 |     43 |        | xmin 1410516400 precedes oldest valid transaction ID 3:1687012112
 179036 |     44 |        | xmin 1410527685 precedes oldest valid transaction ID 3:1687012112
 179036 |     45 |        | xmin 1421269000 precedes oldest valid transaction ID 3:1687012112
 179036 |     46 |        | xmin 1421304247 precedes oldest valid transaction ID 3:1687012112
 179036 |     47 |        | xmin 1421333991 precedes oldest valid transaction ID 3:1687012112
 179036 |     48 |        | xmin 1421365062 precedes oldest valid transaction ID 3:1687012112
 179036 |     49 |        | xmin 1421427152 precedes oldest valid transaction ID 3:1687012112
 179036 |     50 |        | xmin 1421442074 precedes oldest valid transaction ID 3:1687012112
 179036 |     51 |        | xmin 1421462607 precedes oldest valid transaction ID 3:1687012112
 179036 |     52 |        | xmin 1421464665 precedes oldest valid transaction ID 3:1687012112
 179036 |     53 |        | xmin 1421472360 precedes oldest valid transaction ID 3:1687012112
 179036 |     54 |        | xmin 1421479152 precedes oldest valid transaction ID 3:1687012112
 179036 |     55 |        | xmin 1424811032 precedes oldest valid transaction ID 3:1687012112
 179036 |     56 |        | xmin 1432758173 precedes oldest valid transaction ID 3:1687012112
 179036 |     57 |        | xmin 1437607659 precedes oldest valid transaction ID 3:1687012112
 179036 |     58 |        | xmin 1437618864 precedes oldest valid transaction ID 3:1687012112
 179036 |     59 |        | xmin 1437621879 precedes oldest valid transaction ID 3:1687012112
 179036 |     60 |        | xmin 1440619832 precedes oldest valid transaction ID 3:1687012112
 179036 |     61 |        | xmin 1440619912 precedes oldest valid transaction ID 3:1687012112
 179036 |     62 |        | xmin 1442052720 precedes oldest valid transaction ID 3:1687012112
 179036 |     63 |        | xmin 1442052739 precedes oldest valid transaction ID 3:1687012112
 179036 |     64 |        | xmin 1442052794 precedes oldest valid transaction ID 3:1687012112
 179036 |     65 |        | xmin 1442052935 precedes oldest valid transaction ID 3:1687012112
 179036 |     66 |        | xmin 1442052962 precedes oldest valid transaction ID 3:1687012112
 179036 |     67 |        | xmin 1442052967 precedes oldest valid transaction ID 3:1687012112
 179036 |     68 |        | xmin 1442871302 precedes oldest valid transaction ID 3:1687012112
(39 rows)

but only last tuple with (179036,68) is really broken. I can read others.

master

(2024-05-17 14:36:57) prd=# SELECT * FROM page_header(get_raw_page('account_login_history', 179036));
      lsn      │ checksum │ flags │ lower │ upper │ special │ pagesize │ version │ prune_xid
───────────────┼──────────┼───────┼───────┼───────┼─────────┼──────────┼─────────┼───────────
 A576/810F4CE0 │        0 │     4 │   296 │   296 │    8192 │     8192 │       4 │         0
(1 row)


replica
prd_aukro=# SELECT * FROM page_header(get_raw_page('account_login_history', 179036));
      lsn      | checksum | flags | lower | upper | special | pagesize | version | prune_xid
---------------+----------+-------+-------+-------+---------+----------+---------+-----------
 A56C/63979DA0 |        0 |     0 |   296 |   296 |    8192 |     8192 |       4 |         0
(1 row)

master

2024-05-17 14:38:48) prd_aukro=# SELECT * FROM page_checksum(get_raw_page('account_login_history', 179036), 179036);
 page_checksum
───────────────
         17148
(1 row)

replica

prd_aukro=# SELECT * FROM page_checksum(get_raw_page('account_login_history', 179036), 179036);
 page_checksum
---------------
        -17522
(1 row)

The server was under load - but the related tuples was not changed

master

(2024-05-17 14:41:35) prd=# SELECT * FROM heap_page_items(get_raw_page('account_login_history', 179036)) where lp = 68;
─[ RECORD 1 ]───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
lp          │ 68
lp_off      │ 296
lp_flags    │ 1
lp_len      │ 92
t_xmin      │ 1442871302
t_xmax      │ 0
t_field3    │ 0
t_ctid      │ (179036,68)
t_infomask2 │ 9
t_infomask  │ 2819
t_hoff      │ 32
t_bits      │ 1111110100000000
t_oid       │ ∅


replica

prd=# SELECT * FROM heap_page_items(get_raw_page('account_login_history', 179036)) where lp = 68;
-[ RECORD 1 ]---------------------------------------------------------------------------------------------------------------------------
lp          | 68
lp_off      | 296
lp_flags    | 1
lp_len      | 92
t_xmin      | 1442871302
t_xmax      | 0
t_field3    | 0
t_ctid      | (179036,68)
t_infomask2 | 9
t_infomask  | 2051
t_hoff      | 32
t_bits      | 1111110100000000
t_oid       |

master

(2024-05-17 14:45:30) prd=# SELECT t_ctid, raw_flags, combined_flags
         FROM heap_page_items(get_raw_page('account_login_history', 179036)),
           LATERAL heap_tuple_infomask_flags(t_infomask, t_infomask2)
         WHERE t_infomask IS NOT NULL OR t_infomask2 IS NOT NULL;
─[ RECORD 1 ]──┬────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,1)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 2 ]──┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,2)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 3 ]──┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,3)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 4 ]──┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,4)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 5 ]──┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,5)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 6 ]──┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,6)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 7 ]──┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,7)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 8 ]──┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,8)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 9 ]──┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,9)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 10 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,10)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 11 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,11)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 12 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,12)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 13 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,13)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 14 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,14)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 15 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,15)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 16 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,16)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 17 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,17)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 18 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,18)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 19 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,19)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 20 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,20)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 21 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,21)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 22 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,22)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 23 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,23)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 24 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,24)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 25 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,25)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 26 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,26)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 27 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,27)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 28 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,28)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 29 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,29)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 30 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,30)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 31 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,31)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 32 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,32)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 33 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,33)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 34 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,34)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 35 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,35)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 36 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,36)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 37 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,37)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 38 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,38)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 39 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,39)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 40 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,40)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 41 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,41)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 42 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,42)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 43 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,43)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 44 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,44)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 45 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,45)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 46 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,46)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 47 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,47)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 48 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,48)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 49 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,49)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 50 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,50)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 51 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,51)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 52 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,52)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 53 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,53)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 54 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,54)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 55 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,55)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 56 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,56)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 57 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,57)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 58 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,58)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 59 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,59)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 60 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,60)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 61 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,61)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 62 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,62)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 63 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,63)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 64 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,64)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 65 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,65)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 66 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,66)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 67 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,67)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}
─[ RECORD 68 ]─┼────────────────────────────────────────────────────────────────────────────────────────
t_ctid         │ (179036,68)
raw_flags      │ {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags │ {HEAP_XMIN_FROZEN}

replica

prd=# SELECT t_ctid, raw_flags, combined_flags
prd-#          FROM heap_page_items(get_raw_page('account_login_history', 179036)),
prd-#            LATERAL heap_tuple_infomask_flags(t_infomask, t_infomask2)
prd-#          WHERE t_infomask IS NOT NULL OR t_infomask2 IS NOT NULL;
-[ RECORD 1 ]--+----------------------------------------------------------------------------------------
t_ctid         | (179036,1)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 2 ]--+----------------------------------------------------------------------------------------
t_ctid         | (179036,2)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 3 ]--+----------------------------------------------------------------------------------------
t_ctid         | (179036,3)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 4 ]--+----------------------------------------------------------------------------------------
t_ctid         | (179036,4)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 5 ]--+----------------------------------------------------------------------------------------
t_ctid         | (179036,5)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 6 ]--+----------------------------------------------------------------------------------------
t_ctid         | (179036,6)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 7 ]--+----------------------------------------------------------------------------------------
t_ctid         | (179036,7)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 8 ]--+----------------------------------------------------------------------------------------
t_ctid         | (179036,8)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 9 ]--+----------------------------------------------------------------------------------------
t_ctid         | (179036,9)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 10 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,10)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 11 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,11)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 12 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,12)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 13 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,13)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 14 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,14)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 15 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,15)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 16 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,16)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 17 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,17)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 18 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,18)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 19 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,19)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 20 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,20)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 21 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,21)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 22 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,22)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 23 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,23)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 24 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,24)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 25 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,25)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 26 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,26)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 27 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,27)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 28 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,28)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 29 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,29)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
combined_flags | {HEAP_XMIN_FROZEN}
-[ RECORD 30 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,30)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 31 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,31)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 32 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,32)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 33 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,33)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 34 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,34)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 35 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,35)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 36 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,36)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 37 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,37)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 38 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,38)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 39 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,39)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 40 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,40)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 41 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,41)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 42 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,42)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 43 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,43)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 44 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,44)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 45 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,45)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 46 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,46)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 47 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,47)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 48 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,48)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 49 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,49)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 50 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,50)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 51 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,51)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 52 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,52)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 53 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,53)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 54 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,54)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 55 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,55)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 56 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,56)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 57 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,57)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 58 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,58)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 59 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,59)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 60 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,60)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 61 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,61)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 62 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,62)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 63 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,63)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 64 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,64)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 65 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,65)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 66 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,66)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 67 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,67)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
combined_flags | {}
-[ RECORD 68 ]-+----------------------------------------------------------------------------------------
t_ctid         | (179036,68)
raw_flags      | {HEAP_HASNULL,HEAP_HASVARWIDTH,HEAP_XMAX_INVALID}
combined_flags | {}

regards

prd=# select version();
-[ RECORD 1 ]----------------------------------------------------------------------------------------------------
version | PostgreSQL 16.2 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20), 64-bit

Pavel

Re: broken tables on hot standby after migration on PostgreSQL 16 (3x times last month)

From
Peter Geoghegan
Date:
On Fri, May 17, 2024 at 9:13 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> after migration on PostgreSQL 16 I seen 3x times (about every week) broken tables on replica nodes. The query fails
witherror 
>
> ERROR:  could not access status of transaction 1442871302
> DETAIL:  Could not open file "pg_xact/0560": No such file or directory

You've shown an inconsistency between the primary and standby with
respect to the heap tuple infomask bits related to freezing. It looks
like a FREEZE WAL record from the primary was never replayed on the
standby.

It's natural for me to wonder if my Postgres 16 work on page-level
freezing might be a factor here. If that really was true, then it
would be necessary to explain why the primary and standby are
inconsistent (no reason to suspect a problem on the primary here).
It'd have to be the kind of issue that could be detected mechanically
using wal_consistency_checking, but wasn't detected that way before
now -- that seems unlikely.

It's worth considering if the more aggressive behavior around
relfrozenxid advancement (in 15) and freezing (in 16) has increased
the likelihood of problems like these in setups that were already
faulty, in whatever way. The standby database is indeed corrupt, but
even on 16 it's fairly isolated corruption in practical terms. The
full extent of the problem is clear once amcheck is run, but only one
tuple can actually cause the system to error due to the influence of
hint bits (for better or worse, hint bits mask the problem quite well,
even on 16).

--
Peter Geoghegan

Re: broken tables on hot standby after migration on PostgreSQL 16 (3x times last month)

From
Pavel Stehule
Date:


pá 17. 5. 2024 v 18:02 odesílatel Peter Geoghegan <pg@bowt.ie> napsal:
On Fri, May 17, 2024 at 9:13 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> after migration on PostgreSQL 16 I seen 3x times (about every week) broken tables on replica nodes. The query fails with error
>
> ERROR:  could not access status of transaction 1442871302
> DETAIL:  Could not open file "pg_xact/0560": No such file or directory

You've shown an inconsistency between the primary and standby with
respect to the heap tuple infomask bits related to freezing. It looks
like a FREEZE WAL record from the primary was never replayed on the
standby.

It think is possible so broken tuples was created before upgrade from Postgres 15 to Postgres 16 - not too far before, so this bug can be side effect of upgrade

 

It's natural for me to wonder if my Postgres 16 work on page-level
freezing might be a factor here. If that really was true, then it
would be necessary to explain why the primary and standby are
inconsistent (no reason to suspect a problem on the primary here).
It'd have to be the kind of issue that could be detected mechanically
using wal_consistency_checking, but wasn't detected that way before
now -- that seems unlikely.

It's worth considering if the more aggressive behavior around
relfrozenxid advancement (in 15) and freezing (in 16) has increased
the likelihood of problems like these in setups that were already
faulty, in whatever way. The standby database is indeed corrupt, but
even on 16 it's fairly isolated corruption in practical terms. The
full extent of the problem is clear once amcheck is run, but only one
tuple can actually cause the system to error due to the influence of
hint bits (for better or worse, hint bits mask the problem quite well,
even on 16).

--
Peter Geoghegan

Re: broken tables on hot standby after migration on PostgreSQL 16 (3x times last month)

From
Peter Geoghegan
Date:
On Fri, May 17, 2024 at 1:18 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> pá 17. 5. 2024 v 18:02 odesílatel Peter Geoghegan <pg@bowt.ie> napsal:
>> You've shown an inconsistency between the primary and standby with
>> respect to the heap tuple infomask bits related to freezing. It looks
>> like a FREEZE WAL record from the primary was never replayed on the
>> standby.
>
>
> It think is possible so broken tuples was created before upgrade from Postgres 15 to Postgres 16 - not too far
before,so this bug can be side effect of upgrade 

I half suspected something like that myself. Maybe the problem
happened *during* the upgrade, even.

There have been historical bugs affecting pg_upgrade and relfrozenxid.
Commit 74cf7d46 is one good example from only a few years ago.

--
Peter Geoghegan

Re: broken tables on hot standby after migration on PostgreSQL 16 (3x times last month)

From
Andres Freund
Date:
Hi,

On 2024-05-17 15:12:31 +0200, Pavel Stehule wrote:
> after migration on PostgreSQL 16 I seen 3x times (about every week) broken
> tables on replica nodes. The query fails with error

Migrating from what version?


You're saying that the data is correctly accessible on primaries, but broken
on standbys? Is there any difference in how the page looks like on the primary
vs standby?


> ERROR:  could not access status of transaction 1442871302
> DETAIL:  Could not open file "pg_xact/0560": No such file or directory
>
> verify_heapam reports
>
> ^[[Aprd=# select * from verify_heapam('account_login_history') where blkno
> = 179036;
>  blkno  | offnum | attnum |                                msg
>
> --------+--------+--------+-------------------------------------------------------------------
>  179036 |     30 |        | xmin 1393743382 precedes oldest valid
> transaction ID 3:1687012112

So that's not just a narrow race...


> master
>
> (2024-05-17 14:36:57) prd=# SELECT * FROM
> page_header(get_raw_page('account_login_history', 179036));
>       lsn      │ checksum │ flags │ lower │ upper │ special │ pagesize │
> version │ prune_xid
> ───────────────┼──────────┼───────┼───────┼───────┼─────────┼──────────┼─────────┼───────────
>  A576/810F4CE0 │        0 │     4 │   296 │   296 │    8192 │     8192 │
>     4 │         0
> (1 row)
>
>
> replica
> prd_aukro=# SELECT * FROM page_header(get_raw_page('account_login_history',
> 179036));
>       lsn      | checksum | flags | lower | upper | special | pagesize |
> version | prune_xid
> ---------------+----------+-------+-------+-------+---------+----------+---------+-----------
>  A56C/63979DA0 |        0 |     0 |   296 |   296 |    8192 |     8192 |
>     4 |         0
> (1 row)

Is the replica behind the primary? Or did we somehow end up with diverging
data? The page LSNs differ by about 40GB...

Is there evidence of failed truncations of the relation in the log? From
autovacuum?

Does the data in the readable versions of the tuples on that page actually
look valid? Is it possibly duplicated data?


I'm basically wondering whether it's possible that we errored out during
truncation (e.g. due to a file permission issue or such). Due to some
brokenness in RelationTruncate() that can lead to data divergence between
primary and standby and to old tuples re-appearing on either.


Another question: Do you use pg_repack or such?

Greetings,

Andres Freund

Re: broken tables on hot standby after migration on PostgreSQL 16 (3x times last month)

From
Peter Geoghegan
Date:
On Fri, May 17, 2024 at 3:50 PM Andres Freund <andres@anarazel.de> wrote:
> You're saying that the data is correctly accessible on primaries, but broken
> on standbys? Is there any difference in how the page looks like on the primary
> vs standby?

There clearly is. The relevant infomask bits are different. I didn't
examine it much closer than that, though.

--
Peter Geoghegan

Re: broken tables on hot standby after migration on PostgreSQL 16 (3x times last month)

From
Pavel Stehule
Date:


pá 17. 5. 2024 v 21:50 odesílatel Andres Freund <andres@anarazel.de> napsal:
Hi,

On 2024-05-17 15:12:31 +0200, Pavel Stehule wrote:
> after migration on PostgreSQL 16 I seen 3x times (about every week) broken
> tables on replica nodes. The query fails with error

Migrating from what version?

I think 14, but it should be verified tomorrow 


You're saying that the data is correctly accessible on primaries, but broken
on standbys? Is there any difference in how the page looks like on the primary
vs standby?

I saved one page from master and standby. Can I send it privately? There are some private data (although not too sensitive)
 


> ERROR:  could not access status of transaction 1442871302
> DETAIL:  Could not open file "pg_xact/0560": No such file or directory
>
> verify_heapam reports
>
> ^[[Aprd=# select * from verify_heapam('account_login_history') where blkno
> = 179036;
>  blkno  | offnum | attnum |                                msg
>
> --------+--------+--------+-------------------------------------------------------------------
>  179036 |     30 |        | xmin 1393743382 precedes oldest valid
> transaction ID 3:1687012112

So that's not just a narrow race...


> master
>
> (2024-05-17 14:36:57) prd=# SELECT * FROM
> page_header(get_raw_page('account_login_history', 179036));
>       lsn      │ checksum │ flags │ lower │ upper │ special │ pagesize │
> version │ prune_xid
> ───────────────┼──────────┼───────┼───────┼───────┼─────────┼──────────┼─────────┼───────────
>  A576/810F4CE0 │        0 │     4 │   296 │   296 │    8192 │     8192 │
>     4 │         0
> (1 row)
>
>
> replica
> prd_aukro=# SELECT * FROM page_header(get_raw_page('account_login_history',
> 179036));
>       lsn      | checksum | flags | lower | upper | special | pagesize |
> version | prune_xid
> ---------------+----------+-------+-------+-------+---------+----------+---------+-----------
>  A56C/63979DA0 |        0 |     0 |   296 |   296 |    8192 |     8192 |
>     4 |         0
> (1 row)

Is the replica behind the primary? Or did we somehow end up with diverging
data? The page LSNs differ by about 40GB...

Is there evidence of failed truncations of the relation in the log? From
autovacuum?

no I did not see these bugs,

Does the data in the readable versions of the tuples on that page actually
look valid? Is it possibly duplicated data?

looks well, I didn't see any strange content


I'm basically wondering whether it's possible that we errored out during
truncation (e.g. due to a file permission issue or such). Due to some
brokenness in RelationTruncate() that can lead to data divergence between
primary and standby and to old tuples re-appearing on either.


Another question: Do you use pg_repack or such?

pg_repack was used 2 months before migration 

 

Greetings,

Andres Freund

Re: broken tables on hot standby after migration on PostgreSQL 16 (3x times last month)

From
Pavel Stehule
Date:


pá 17. 5. 2024 v 22:05 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:


pá 17. 5. 2024 v 21:50 odesílatel Andres Freund <andres@anarazel.de> napsal:
Hi,

On 2024-05-17 15:12:31 +0200, Pavel Stehule wrote:
> after migration on PostgreSQL 16 I seen 3x times (about every week) broken
> tables on replica nodes. The query fails with error

Migrating from what version?

I think 14, but it should be verified tomorrow 

upgrade was from 15.2

 


You're saying that the data is correctly accessible on primaries, but broken
on standbys? Is there any difference in how the page looks like on the primary
vs standby?

I saved one page from master and standby. Can I send it privately? There are some private data (although not too sensitive)
 


> ERROR:  could not access status of transaction 1442871302
> DETAIL:  Could not open file "pg_xact/0560": No such file or directory
>
> verify_heapam reports
>
> ^[[Aprd=# select * from verify_heapam('account_login_history') where blkno
> = 179036;
>  blkno  | offnum | attnum |                                msg
>
> --------+--------+--------+-------------------------------------------------------------------
>  179036 |     30 |        | xmin 1393743382 precedes oldest valid
> transaction ID 3:1687012112

So that's not just a narrow race...


> master
>
> (2024-05-17 14:36:57) prd=# SELECT * FROM
> page_header(get_raw_page('account_login_history', 179036));
>       lsn      │ checksum │ flags │ lower │ upper │ special │ pagesize │
> version │ prune_xid
> ───────────────┼──────────┼───────┼───────┼───────┼─────────┼──────────┼─────────┼───────────
>  A576/810F4CE0 │        0 │     4 │   296 │   296 │    8192 │     8192 │
>     4 │         0
> (1 row)
>
>
> replica
> prd_aukro=# SELECT * FROM page_header(get_raw_page('account_login_history',
> 179036));
>       lsn      | checksum | flags | lower | upper | special | pagesize |
> version | prune_xid
> ---------------+----------+-------+-------+-------+---------+----------+---------+-----------
>  A56C/63979DA0 |        0 |     0 |   296 |   296 |    8192 |     8192 |
>     4 |         0
> (1 row)

Is the replica behind the primary? Or did we somehow end up with diverging
data? The page LSNs differ by about 40GB...

Is there evidence of failed truncations of the relation in the log? From
autovacuum?

no I did not see these bugs,

Does the data in the readable versions of the tuples on that page actually
look valid? Is it possibly duplicated data?

looks well, I didn't see any strange content


I'm basically wondering whether it's possible that we errored out during
truncation (e.g. due to a file permission issue or such). Due to some
brokenness in RelationTruncate() that can lead to data divergence between
primary and standby and to old tuples re-appearing on either.


Another question: Do you use pg_repack or such?

pg_repack was used 2 months before migration 

 

Greetings,

Andres Freund

Re: broken tables on hot standby after migration on PostgreSQL 16 (3x times last month)

From
Pavel Stehule
Date:
Hi




Another question: Do you use pg_repack or such?

pg_repack was used for some tables, but I found broken tables, where pg_repack was not used.

Regards

Pavel

Greetings,

Andres Freund

Re: broken tables on hot standby after migration on PostgreSQL 16 (3x times last month)

From
Andres Freund
Date:
On 2024-05-17 16:03:09 -0400, Peter Geoghegan wrote:
> On Fri, May 17, 2024 at 3:50 PM Andres Freund <andres@anarazel.de> wrote:
> > You're saying that the data is correctly accessible on primaries, but broken
> > on standbys? Is there any difference in how the page looks like on the primary
> > vs standby?
> 
> There clearly is. The relevant infomask bits are different. I didn't
> examine it much closer than that, though.

That could also just be because of a different replay position, hence my
question about that somewhere else in the email...