This might be due to the upgrade in glibc between RHEL7 and RHEL8, which includes updated collations that change sort orders and thus invalidate existing indexes, forcing them to be rebuilt (on a RHEL>7 system) if they contain any values which sort differently under the new collation than the old one.
We are facing an issue related to index corruption after migrating our PostgreSQL 11 setup from RHEL 7 to RHEL 9 using streaming replication (base backup method).
After bringing up the standby on RHEL 9, we observed that certain tables are not returning results when queries use indexed scans. Upon investigation, the following check confirms index corruption:
The same indexes work fine on the RHEL 7 (primary) side. However, on the RHEL 9 replica, queries that rely on this index return zero records.
Rebuilding the indexes fixed the issue temporarily but we have many indexes and our DB size is more than 10TB.
Environment details:
PostgreSQL Version: 11.15
OS on primary: RHEL 7.9
OS on standby: RHEL 9.6
Replication Type: Streaming replication (initialized using pg_basebackup)
Data Directory initialized from RHEL 7 base backup
Issue Summary:
Indexes appear and are the same size as per prod on standby after base backup restore.
We are able to read the data from the tables without index scans on standy by RHEL 9.
No filesystem or WAL errors observed in logs.
Could this be related to OS-level binary or page layout differences between RHEL 7 and RHEL 9 for PostgreSQL 11 binaries? Any insights or recommended actions would be greatly appreciated.
