Thread: pg_upgrade fails, "mismatch of relation OID" - 9.1.9 to 9.2.4
I've tried several times to upgrade a test database (with real data, ~500 GB) from 9.1 to 9.2 using pg_upgrade and every time it fails with the same error. I've tried a few different options to pg_upgrade but always the same result. Nothing really useful has turned up in Google. Any thoughts? Complete output is below: -bash-4.1$ time /usr/pgsql-9.2/bin/pg_upgrade -b /usr/pgsql-9.1/bin/ -B /usr/pgsql-9.2/bin/ -d /var/lib/pgsql/9.1/data/ -D /var/lib/pgsql/9.2/data/ -P 50433 --link Performing Consistency Checks ----------------------------- Checking current, bin, and data directories ok Checking cluster versions ok Checking database user is a superuser ok Checking for prepared transactions ok Checking for reg* system OID user data types ok Checking for contrib/isn with bigint-passing mismatch ok Creating catalog dump ok Checking for presence of required libraries ok Checking database user is a superuser ok Checking for prepared transactions ok If pg_upgrade fails after this point, you must re-initdb the new cluster before continuing. Performing Upgrade ------------------ Analyzing all rows in the new cluster ok Freezing all rows on the new cluster ok Deleting files from new pg_clog ok Copying old pg_clog to new server ok Setting next transaction ID for new cluster ok Resetting WAL archives ok Setting frozenxid counters in new cluster ok Creating databases in the new cluster ok Adding support functions to new cluster ok Restoring database schema to new cluster ok Removing support functions from new cluster ok Adding ".old" suffix to old global/pg_control ok If you want to start the old cluster, you will need to remove the ".old" suffix from /var/lib/pgsql/9.1/data/global/pg_control.old. Because "link" mode was used, the old cluster cannot be safely started once the new cluster has been started. Linking user relation files /var/lib/pgsql/9.1/data/base/16406/3016054 Mismatch of relation OID in database "dbname": old OID 2938685, new OID 299721 Failure, exiting real 12m31.600s user 1m11.594s sys 1m2.519s
> -----Original Message----- > From: pgsql-general-owner@postgresql.org [mailto:pgsql-general- > owner@postgresql.org] On Behalf Of Evan D. Hoffman > Sent: Wednesday, May 08, 2013 2:27 PM > To: Postgresql Mailing List > Subject: [GENERAL] pg_upgrade fails, "mismatch of relation OID" - 9.1.9 > to 9.2.4 > > I've tried several times to upgrade a test database (with real data, > ~500 GB) from 9.1 to 9.2 using pg_upgrade and every time it fails with > the same error. I've tried a few different options to pg_upgrade but > always the same result. Nothing really useful has turned up in Google. > Any thoughts? Complete output is below: > > > > Linking user relation files > /var/lib/pgsql/9.1/data/base/16406/3016054 > Mismatch of relation OID in database "dbname": old OID 2938685, new OID > 299721 Failure, exiting > > Is it always the same file, same OIDs (old/new)? If it's the same file, did you try to find out what relation it belongs to? Igor Neyman
> -----Original Message-----
> From: Evan D. Hoffman [mailto:evandhoffman@gmail.com]
> Sent: Wednesday, May 08, 2013 3:35 PM
> To: Igor Neyman
> Subject: Re: [GENERAL] pg_upgrade fails, "mismatch of relation OID" -
> 9.1.9 to 9.2.4
>
> Looks like it IS the same OID every time, referencing an index. I
> already reindexed the entire DB in case it was some problem with a
> corrupt index. Here's the index info, if it's of any use.
>
>
> Interestingly, if I query which that relation's in, it's not the one
> that it complained about:
>
> db=# select pg_relation_filepath(2938685); pg_relation_filepath
> ----------------------
> base/16407/21446253
> (1 row)
>
> db=#
>
> (The file referenced in the error was
> /var/lib/pgsql/9.1/data/base/16406/3016054)
>
> On Wed, May 8, 2013 at 2:35 PM, Igor Neyman <ineyman@perceptron.com>
> wrote:
> >
> >> -----Original Message-----
> >> From: pgsql-general-owner@postgresql.org [mailto:pgsql-general-
> >> owner@postgresql.org] On Behalf Of Evan D. Hoffman
> >> Sent: Wednesday, May 08, 2013 2:27 PM
> >> To: Postgresql Mailing List
> >> Subject: [GENERAL] pg_upgrade fails, "mismatch of relation OID" -
> >> 9.1.9 to 9.2.4
> >>
> >>
> >> Linking user relation files
> >> /var/lib/pgsql/9.1/data/base/16406/3016054
> >> Mismatch of relation OID in database "dbname": old OID 2938685, new
> >> OID
> >> 299721 Failure, exiting
> >>
> >>
> >
> > Is it always the same file, same OIDs (old/new)?
> > If it's the same file, did you try to find out what relation it
> belongs to?
> >
> > Igor Neyman
> >
Is it the same file though?
And, if it is what do you get when you run:
Select relname from pg_class where relfilenode = 3016054::oid;
Please, reply to the list ("reply to all"), so that other people who may have better ideas/solutions for could see it.
Igor Neyman
Well, each time it fails it refers to the file
"/var/lib/pgsql/9.1/data/base/16406/3016054", but that's not the file
associated with OID 2938685.
Here's the output of that query:
db=# Select relname from pg_class where relfilenode = 3016054::oid;
relname
---------
(0 rows)
db=#
On Wed, May 8, 2013 at 4:12 PM, Igor Neyman <ineyman@perceptron.com> wrote:
>
>
>> -----Original Message-----
>> From: Evan D. Hoffman [mailto:evandhoffman@gmail.com]
>> Sent: Wednesday, May 08, 2013 3:35 PM
>> To: Igor Neyman
>> Subject: Re: [GENERAL] pg_upgrade fails, "mismatch of relation OID" -
>> 9.1.9 to 9.2.4
>>
>> Looks like it IS the same OID every time, referencing an index. I
>> already reindexed the entire DB in case it was some problem with a
>> corrupt index. Here's the index info, if it's of any use.
>>
>>
>> Interestingly, if I query which that relation's in, it's not the one
>> that it complained about:
>>
>> db=# select pg_relation_filepath(2938685); pg_relation_filepath
>> ----------------------
>> base/16407/21446253
>> (1 row)
>>
>> db=#
>>
>> (The file referenced in the error was
>> /var/lib/pgsql/9.1/data/base/16406/3016054)
>>
>> On Wed, May 8, 2013 at 2:35 PM, Igor Neyman <ineyman@perceptron.com>
>> wrote:
>> >
>> >> -----Original Message-----
>> >> From: pgsql-general-owner@postgresql.org [mailto:pgsql-general-
>> >> owner@postgresql.org] On Behalf Of Evan D. Hoffman
>> >> Sent: Wednesday, May 08, 2013 2:27 PM
>> >> To: Postgresql Mailing List
>> >> Subject: [GENERAL] pg_upgrade fails, "mismatch of relation OID" -
>> >> 9.1.9 to 9.2.4
>> >>
>> >>
>> >> Linking user relation files
>> >> /var/lib/pgsql/9.1/data/base/16406/3016054
>> >> Mismatch of relation OID in database "dbname": old OID 2938685, new
>> >> OID
>> >> 299721 Failure, exiting
>> >>
>> >>
>> >
>> > Is it always the same file, same OIDs (old/new)?
>> > If it's the same file, did you try to find out what relation it
>> belongs to?
>> >
>> > Igor Neyman
>> >
>
> Is it the same file though?
> And, if it is what do you get when you run:
>
> Select relname from pg_class where relfilenode = 3016054::oid;
>
> Please, reply to the list ("reply to all"), so that other people who may have better ideas/solutions for could see
it.
>
> Igor Neyman
"Evan D. Hoffman" <evandhoffman@gmail.com> writes:
> (The file referenced in the error was
> /var/lib/pgsql/9.1/data/base/16406/3016054)
I'm not sure about how pg_upgrade manages its output, but it seems
entirely possible that that was the last file successfully transferred,
not the one the error occurred on.
> Looks like it IS the same OID every time, referencing an index.
What index exactly? Anything different about that index (or its
table) from others in the database?
regards, tom lane
> -----Original Message----- > From: Evan D. Hoffman [mailto:evandhoffman@gmail.com] > Sent: Wednesday, May 08, 2013 4:22 PM > To: Igor Neyman > Cc: pgsql-general@postgresql.org > Subject: Re: [GENERAL] pg_upgrade fails, "mismatch of relation OID" - > 9.1.9 to 9.2.4 > > Well, each time it fails it refers to the file > "/var/lib/pgsql/9.1/data/base/16406/3016054", but that's not the file > associated with OID 2938685. > > Here's the output of that query: > > db=# Select relname from pg_class where relfilenode = 3016054::oid; > relname > --------- > (0 rows) > > db=# > > And that is before running pg_upgrade, right? Seems like some kind of pg_catalog corruption. I guess, Bruce Momjian would know better, what's going on here. Igor Neyman