Thread: Problem with pg_upgrade 8.3 to 9.1.4 - clog missing?!
hi I just upgraded test copy of database of our customer (~ 600GB of data). upgrade went fine, no errors. but vacuumdb -azv ended with an error: => vacuumdb --all --analyze -p 6665 vacuumdb: vacuuming database "client_db" vacuumdb: vacuuming database "pg_audit" vacuumdb: vacuuming database "postgres" vacuumdb: vacuuming of database "postgres" failed: ERROR: could not access status of transaction 860626316 DETAIL: Could not open file "pg_clog/0334": No such file or directory. I know we had these kind of errors before, but I thought they all got fixed. What can I do to debug it more? 8.3 was 8.3.16. 8.3.16 is from rpms, 9.1.4 is from source compilation. both have integer datatimes, and both are for 64bit linux. pg_clog contains: -rw------- 1 postgres postgres 262144 Jun 6 21:22 04A4 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04A5 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04A6 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04A7 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04A8 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04A9 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04AA -rw------- 1 postgres postgres 262144 Jun 6 21:22 04AB -rw------- 1 postgres postgres 262144 Jun 6 21:22 04AC -rw------- 1 postgres postgres 262144 Jun 6 21:22 04AD -rw------- 1 postgres postgres 262144 Jun 6 21:22 04AE -rw------- 1 postgres postgres 262144 Jun 6 21:22 04AF -rw------- 1 postgres postgres 262144 Jun 6 21:22 04B0 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04B1 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04B2 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04B3 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04B4 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04B5 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04B6 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04B7 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04B8 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04B9 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04BA -rw------- 1 postgres postgres 262144 Jun 6 21:22 04BB -rw------- 1 postgres postgres 262144 Jun 6 21:22 04BC -rw------- 1 postgres postgres 262144 Jun 6 21:22 04BD -rw------- 1 postgres postgres 262144 Jun 6 21:22 04BE -rw------- 1 postgres postgres 262144 Jun 6 21:22 04BF -rw------- 1 postgres postgres 262144 Jun 6 21:22 04C0 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04C1 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04C2 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04C3 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04C4 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04C5 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04C6 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04C7 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04C8 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04C9 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04CA -rw------- 1 postgres postgres 262144 Jun 6 21:22 04CB -rw------- 1 postgres postgres 262144 Jun 6 21:22 04CC -rw------- 1 postgres postgres 262144 Jun 6 21:22 04CD -rw------- 1 postgres postgres 262144 Jun 6 21:22 04CE -rw------- 1 postgres postgres 262144 Jun 6 21:22 04CF -rw------- 1 postgres postgres 262144 Jun 6 21:22 04D0 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04D1 -rw------- 1 postgres postgres 262144 Jun 6 21:22 04D2 -rw------- 1 postgres postgres 196608 Jun 7 00:15 04D3 unfortunately i do not have the 8.3 data dir any more. if the problem is only with postgres database - i can ignore it. but I'd rather be sure what is the source of the issue. Best regards, depesz -- The best thing about modern society is how easy it is to avoid contact with it. http://depesz.com/
On Thu, Jun 07, 2012 at 09:16:49PM +0200, hubert depesz lubaczewski wrote: > hi > I just upgraded test copy of database of our customer (~ 600GB of data). > upgrade went fine, no errors. but vacuumdb -azv ended with an error: > > => vacuumdb --all --analyze -p 6665 > vacuumdb: vacuuming database "client_db" > vacuumdb: vacuuming database "pg_audit" > vacuumdb: vacuuming database "postgres" > vacuumdb: vacuuming of database "postgres" failed: ERROR: could not access status of transaction 860626316 > DETAIL: Could not open file "pg_clog/0334": No such file or directory. > > I know we had these kind of errors before, but I thought they all got fixed. > > What can I do to debug it more? > > 8.3 was 8.3.16. > 8.3.16 is from rpms, 9.1.4 is from source compilation. > > both have integer datatimes, and both are for 64bit linux. I assume 8.3 vacuumdb did not generate these errors. If it was 8.4 I would suggest it was because we don't copy visibility map files from pre-9.1 because those were not crash-safe, but 8.3 didn't have visibility map files (added in PG 8.4), so 8.3 should have been checking the exact same transaction ids as 9.1. Basically, I can't think of a cause. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On Wed, Jun 13, 2012 at 02:32:21PM -0400, Bruce Momjian wrote: > I assume 8.3 vacuumdb did not generate these errors. If it was 8.4 I > would suggest it was because we don't copy visibility map files from > pre-9.1 because those were not crash-safe, but 8.3 didn't have > visibility map files (added in PG 8.4), so 8.3 should have been checking > the exact same transaction ids as 9.1. > > Basically, I can't think of a cause. yeah, neither can we. but - any other test - even using the same source of $PGDATA went flawless. I'm tempted to assume it's just a random glitch in matrix, and just go on. Best regards, depesz -- The best thing about modern society is how easy it is to avoid contact with it. http://depesz.com/
On Wed, Jun 13, 2012 at 08:43:23PM +0200, hubert depesz lubaczewski wrote: > On Wed, Jun 13, 2012 at 02:32:21PM -0400, Bruce Momjian wrote: > > I assume 8.3 vacuumdb did not generate these errors. If it was 8.4 I > > would suggest it was because we don't copy visibility map files from > > pre-9.1 because those were not crash-safe, but 8.3 didn't have > > visibility map files (added in PG 8.4), so 8.3 should have been checking > > the exact same transaction ids as 9.1. > > > > Basically, I can't think of a cause. > > yeah, neither can we. but - any other test - even using the same source > of $PGDATA went flawless. I'm tempted to assume it's just a random > glitch in matrix, and just go on. Is the vacuumdb still repeatedly failing? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On Wed, Jun 13, 2012 at 02:55:41PM -0400, Bruce Momjian wrote: > On Wed, Jun 13, 2012 at 08:43:23PM +0200, hubert depesz lubaczewski wrote: > > On Wed, Jun 13, 2012 at 02:32:21PM -0400, Bruce Momjian wrote: > > > I assume 8.3 vacuumdb did not generate these errors. If it was 8.4 I > > > would suggest it was because we don't copy visibility map files from > > > pre-9.1 because those were not crash-safe, but 8.3 didn't have > > > visibility map files (added in PG 8.4), so 8.3 should have been checking > > > the exact same transaction ids as 9.1. > > > > > > Basically, I can't think of a cause. > > > > yeah, neither can we. but - any other test - even using the same source > > of $PGDATA went flawless. I'm tempted to assume it's just a random > > glitch in matrix, and just go on. > > Is the vacuumdb still repeatedly failing? well - in this particular upgraded database - i think so. I did run it couple of times, it failed all the times. But this $PGDATA is not longer used - we did another tests, and no other test showed problems - i.e. vacuum worked fine. Best regards, depesz