Hi, a quick update:
- We have discovered that the corruption was present from before libicu update.
- We ran `pg_amcheck --index state_groups_state_type_idx --heapallindexed matrix`, which returned nothing
- We believe that means that (and matches what we see sampling) the index has gained extra entries, i.e. that for a given state group it does return all the relevant rows in the table plus extra rows.
We are also seeing old state groups starting to point at rows that have only just been inserted. For example, querying for 353864583 on the primary it returns that row plus four rows that have been inserted today, but on the backup from last week an index only scan for 353864583 only returns one row. This makes it feel like the corruption is ongoing? Nothing should have modified that state group in the interim (they are generally immutable).
This naively feels like when inserting a new row we sometimes add the row to the index twice: once pointing from the correct state group to the new row, and once from an old state group to the new row?
Are checksums enabled in the instance?
Alas not.
We've also now found that the index on the backup does in fact point to those ctids after all, but they are marked as dead. So at some point between then and when we inserted the new row at that ctid today those entries were marked undead.
Copyright © 2025 Element - All rights reserved. The Element name, logo and device are registered trademarks of New Vector Ltd. Registered number: 10873661. Registered in England and Wales. Registered address: 10 Queen Street Place, London, United Kingdom, EC4R 1AG.
This message is intended for the addressee only and may contain private and confidential information or material which may be privileged. If this message has come to you in error please delete it immediately and do not copy it or show it to any other person.
