Thank you for the feedback, Alvaro.
Unfortunately, the database is no longer "dumpable". We were able to do
a pg_dump yesterday morning (12 hours after the crash + purging the
pg_clog) but if we try one now, we get the following error:
unexpected chunk number 1 (expected 0) for toast value 8282331 in
pg_toast_38651
Looking at our data, there seem to be 6 tables that have corrupt
records. Doing a SELECT * for one of those records, will return a
similar error:
missing chunk number 0 for toast value 8288522 in pg_toast_5572299
What is the best way to go from here? Is tracking down these corrupt
records and deleting them the best / only solution?
Is there a way to determine of there are issues with new data (after the
crash)?
Any help and advice is very much appreciated.
Thanks,
Nick Renders
On 5 Feb 2020, at 12:51, Alvaro Herrera wrote:
> On 2020-Feb-05, Nick Renders wrote:
>
>> Is there anything specific I should check in our postgres
>> installation /
>> database to make sure it is running ok now? Anyway to see what the
>> consequences were of purging that one pg_clog file?
>
> Losing pg_clog files is pretty bad, and should not happen; then again,
> this might have been something else (ie. the file was maybe not lost).
> That said, wrongly overwriting files is even worse.
>
> By zeroing an existing pg_clog file, you marked a bunch of
> transactions
> as aborted. Your data is now probably inconsistent, if not downright
> corrupt. I would be looking for my most recent backup ...
>
> If you're very lucky, your database might be pg_dumpable. I would try
> that, followed by restoring it in a separate clean instance and seeing
> what happens.
>
> --
> Álvaro Herrera https://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
