Thread: Upgrading postgresql-8.4
I have a database that's about 600 GB. I did a pg_dump (pg_dump --format plain --create --encoding UTF8 --oids --verbose --file "/var/tmp/145_backup.sql" "digi") and, after 442 GB, I got an error, "pg_dump: Error message from server: ERROR: missing chunk number 0 for toast value 49209130 in pg_toast_17031". This was using postgresql-8.4.3 on Ubuntu 10.04. I upgraded our postgres to 8.4.16 using 'apt-get install postgresql-8.4'.
This went well and postgres restarted just fine. However, now when I execute a pg_dump I get a missing chunk 0 for pg_toast_2619 while querying pg_attribute. I did a reindex on pg_toast_2619, then tried to VACUUM ANALYZE pg_attribute but again got the missing chunk 0 error.
Did I miss a step doing the upgrade or recovery attempt, or is the data corrupted? I did the postgres upgrade and pg_dump on another server and it went very smooth so I'm flustered at how the problem seems to have gotten worse.
This went well and postgres restarted just fine. However, now when I execute a pg_dump I get a missing chunk 0 for pg_toast_2619 while querying pg_attribute. I did a reindex on pg_toast_2619, then tried to VACUUM ANALYZE pg_attribute but again got the missing chunk 0 error.
Did I miss a step doing the upgrade or recovery attempt, or is the data corrupted? I did the postgres upgrade and pg_dump on another server and it went very smooth so I'm flustered at how the problem seems to have gotten worse.
Steve Erickson <serickson@digitiliti.com> writes: > This went well and postgres restarted just fine. However, now when I execute a pg_dump I get a missing chunk 0 for pg_toast_2619while querying pg_attribute. I did a reindex on pg_toast_2619, then tried to VACUUM ANALYZE pg_attribute butagain got the missing chunk 0 error. > Did I miss a step doing the upgrade or recovery attempt, or is the > data corrupted? It's corrupt, but fortunately for you, 2619 is pg_statistic which is eminently discardable data. Just truncate pg_statistic and you should be good. If you aren't immediately abandoning the old database, you might want to re-ANALYZE everything to reconstruct the stats. We've seen one or two reports like this before, which makes me think there might be a reproducible bug lurking somewhere around here; but I don't suppose you have a recipe for getting a database into this state ... regards, tom lane
Thanks for the reply. I deleted all rows in pg_statistic and the VACUUM ANALYZE on pg_attribute still fails. I tried toreindex pg_toast_2619 and got an error "could not access status of transaction 1493786085. Could not open file "pg_subtrans/5909":No such file or directory. Sure enough, there is no such file - only 5905. ________________________________________ From: Tom Lane [tgl@sss.pgh.pa.us] Sent: Monday, March 11, 2013 12:10 PM To: Steve Erickson Cc: pgsql-general@postgresql.org Subject: Re: [GENERAL] Upgrading postgresql-8.4 Steve Erickson <serickson@digitiliti.com> writes: > This went well and postgres restarted just fine. However, now when I execute a pg_dump I get a missing chunk 0 for pg_toast_2619while querying pg_attribute. I did a reindex on pg_toast_2619, then tried to VACUUM ANALYZE pg_attribute butagain got the missing chunk 0 error. > Did I miss a step doing the upgrade or recovery attempt, or is the > data corrupted? It's corrupt, but fortunately for you, 2619 is pg_statistic which is eminently discardable data. Just truncate pg_statistic and you should be good. If you aren't immediately abandoning the old database, you might want to re-ANALYZE everything to reconstruct the stats. We've seen one or two reports like this before, which makes me think there might be a reproducible bug lurking somewhere around here; but I don't suppose you have a recipe for getting a database into this state ... regards, tom lane
Steve Erickson <serickson@digitiliti.com> writes: > Thanks for the reply. I deleted all rows in pg_statistic and the VACUUM ANALYZE on pg_attribute still fails. You would probably need to actually truncate pg_statistic (in a standalone backend) to make that safe. If you only want to use DELETE, I'd try just pg_dump at this point rather than trying to make the database fully usable again. > I tried to reindex pg_toast_2619 and got an error "could not access status of transaction 1493786085. Could not open file"pg_subtrans/5909": No such file or directory. Sure enough, there is no such file - only 5905. This suggests that your problems are larger than you've indicated so far. What happened to this machine anyway? regards, tom lane
On 03/11/2013 10:46 AM, Tom Lane wrote: > Steve Erickson <serickson@digitiliti.com> writes: >> Thanks for the reply. I deleted all rows in pg_statistic and the VACUUM ANALYZE on pg_attribute still fails. > > You would probably need to actually truncate pg_statistic (in a > standalone backend) to make that safe. If you only want to use DELETE, > I'd try just pg_dump at this point rather than trying to make the > database fully usable again. > >> I tried to reindex pg_toast_2619 and got an error "could not access status of transaction 1493786085. Could not openfile "pg_subtrans/5909": No such file or directory. Sure enough, there is no such file - only 5905. > > This suggests that your problems are larger than you've indicated so > far. What happened to this machine anyway? Possibly being bit by this?: http://wiki.postgresql.org/wiki/20110408pg_upgrade_fix > > regards, tom lane > > -- Adrian Klaver adrian.klaver@gmail.com