Thread: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

From
Duarte Carreira
Date:

Hello.

 

I’ve trying to upgrade a 9.3 instance to 9.5 using pg_upgrade and facing this issue…

 

I’ve found a similar thread, but going from 8.4 to 9.2:

https://www.postgresql.org/message-id/flat/52C44FA9.2080500%40gmail.com#14ab24f6c94287dd4c435652cb2d77f8

 

This ended up without really pinpointing the problem as the poster did not follow up.

Hopefully I’ll have more luck…

 

I’m posting below more output, here I’ll crop the interesting bits.

(don’t know if this is pertinent for this discussion but Pg_upgrade complains with 2 errors but still executes, and –check turns out ok.

These are:

pg_upgrade: could not start process for command ""D:\Program Files\PostgreSQL\9.

5\bin\pg_upgrade"  -U postgres --jobs=6 -d "D:\Program Files\PostgreSQL\9.3\data

" -b "D:\Program Files\PostgreSQL\9.3\bin" -D "D:\Program Files\PostgreSQL\9.5\d

ata" -B "D:\Program Files\PostgreSQL\9.5\bin"": error code 193

pg_upgrade: could not re-execute with restricted token: error code 3

)

 

What I’m seeing from pg_upgrade:

 

Restoring database schemas in the new cluster

  template1

*failure*

 

Consult the last few lines of "pg_upgrade_dump_16393.log" for

the probable cause of the failure.

Failure, exiting

 

So, looking at the log file, it’s not very long, at the end where it blows up:

pg_restore: creating TYPE "sde.se_coord"

pg_restore: creating SHELL TYPE "sde.se_extent"

pg_restore: creating FUNCTION "sde.st_extent_in("cstring")"

pg_restore: creating FUNCTION "sde.st_extent_out("sde"."se_extent")"

pg_restore: creating FUNCTION "sde.st_extent_recv("internal")"

pg_restore: creating FUNCTION "sde.st_extent_send("sde"."se_extent")"

pg_restore: creating TYPE "sde.se_extent"

pg_restore: creating FUNCTION "sde.st_envelope_in("cstring")"

pg_restore: [archiver (db)] Error while PROCESSING TOC:

pg_restore: [archiver (db)] Error from TOC entry 4712; 1255 1141726 FUNCTION st_envelope_in("cstring") sde

pg_restore: [archiver (db)] could not execute query: ERROR:  duplicate key value violates unique constraint "pg_type_oid_index"

DETAIL:  Key (oid)=(1142573) already exists.

    Command was: CREATE FUNCTION "sde"."st_envelope_in"("cstring") RETURNS "sde"."st_envelope"

    LANGUAGE "c" IMMUTABLE STRICT

    AS 'st_g...

command: "D:\Program Files\PostgreSQL\9.5\bin/pg_restore" --port 50432 --username ^"postgres^" --exit-on-error --verbose --dbname ^"dbname^=postgis^" "pg_upgrade_dump_16393.custom" >> "pg_upgrade_dump_16393.log" 2>&1

 

It says it’s trying to create function sde.st_envelope_in a 2nd time.

 

It seems to me that while creating TYPE sde.se_coord (or is it SHELL TYPE sde.se_extent?) this function was created.

Right afterwards, when creating TYPE sde.se_extent, the same function is also being created, causing the duplicate error…

 

At least that’s my limited understanding…

 

Can anyone please take a look? I can send any file from the pg_upgrade process.

 

Many thanks,

Duarte

 

 

------------------------------------------full outputs-------------------------------------------------

Pg_upgrade output:

C:\work\upgrade_postgresql\testes>"D:\Program Files\PostgreSQL\9.5\bin\pg_upgrad

e" -U postgres --jobs=6 -d "D:\Program Files\PostgreSQL\9.3\data" -b "D:\Program

Files\PostgreSQL\9.3\bin" -D "D:\Program Files\PostgreSQL\9.5\data" -B "D:\Prog

ram Files\PostgreSQL\9.5\bin"

pg_upgrade: could not start process for command ""D:\Program Files\PostgreSQL\9.

5\bin\pg_upgrade"  -U postgres --jobs=6 -d "D:\Program Files\PostgreSQL\9.3\data

" -b "D:\Program Files\PostgreSQL\9.3\bin" -D "D:\Program Files\PostgreSQL\9.5\d

ata" -B "D:\Program Files\PostgreSQL\9.5\bin"": error code 193

pg_upgrade: could not re-execute with restricted token: error code 3

Performing Consistency Checks

-----------------------------

Checking cluster versions                                   ok

Checking database user is the install user                  ok

Checking database connection settings                       ok

Checking for prepared transactions                          ok

Checking for reg* system OID user data types                ok

Checking for contrib/isn with bigint-passing mismatch       ok

Checking for invalid "line" user columns                    ok

Creating dump of global objects                             ok

Creating dump of database schemas

                                                            ok

Checking for presence of required libraries                 ok

Checking database user is the install user                  ok

Checking for prepared transactions                          ok

 

If pg_upgrade fails after this point, you must re-initdb the

new cluster before continuing.

 

Performing Upgrade

------------------

Analyzing all rows in the new cluster                       ok

Freezing all rows on the new cluster                        ok

Deleting files from new pg_clog                             ok

Copying old pg_clog to new server                           ok

Setting next transaction ID and epoch for new cluster       ok

Deleting files from new pg_multixact/offsets                ok

Copying old pg_multixact/offsets to new server              ok

Deleting files from new pg_multixact/members                ok

Copying old pg_multixact/members to new server              ok

Setting next multixact ID and offset for new cluster        ok

Resetting WAL archives                                      ok

Setting frozenxid and minmxid counters in new cluster       ok

Restoring global objects in the new cluster                 ok

Restoring database schemas in the new cluster

  template1

*failure*

 

Consult the last few lines of "pg_upgrade_dump_16393.log" for

the probable cause of the failure.

Failure, exiting

 

 

+++++++++++++++++++++++++++++++++++++++++++++

Log contents:

command: "D:\Program Files\PostgreSQL\9.5\bin/pg_dump" --port 50432 --username ^"postgres^" --schema-only --quote-all-identifiers --binary-upgrade --format=custom  --file="pg_upgrade_dump_16393.custom" ^"dbname^=postgis^" >> "pg_upgrade_dump_16393.log" 2>&1

 

 

pg_restore: connecting to database for restore

pg_restore: creating pg_largeobject "pg_largeobject"

pg_restore: creating pg_largeobject_metadata "pg_largeobject_metadata"

pg_restore: creating SCHEMA "agdalentejo"

pg_restore: creating COMMENT "SCHEMA "agdalentejo""

pg_restore: creating SCHEMA "agdsandre"

pg_restore: creating COMMENT "SCHEMA "agdsandre""

pg_restore: creating SCHEMA "ags_rw"

pg_restore: creating SCHEMA "algas"

pg_restore: creating SCHEMA "always"

pg_restore: creating SCHEMA "ciefma"

pg_restore: creating SCHEMA "dc"

pg_restore: creating SCHEMA "dcom"

pg_restore: creating SCHEMA "dgaf"

pg_restore: creating SCHEMA "diap"

pg_restore: creating COMMENT "SCHEMA "diap""

pg_restore: creating SCHEMA "dpci"

pg_restore: creating SCHEMA "dpea"

pg_restore: creating COMMENT "SCHEMA "dpea""

pg_restore: creating SCHEMA "dsi"

pg_restore: creating SCHEMA "edia"

pg_restore: creating SCHEMA "gdbman"

pg_restore: creating COMMENT "SCHEMA "gdbman""

pg_restore: creating SCHEMA "glc"

pg_restore: creating SCHEMA "navia"

pg_restore: creating SCHEMA "portaldados"

pg_restore: creating SCHEMA "public"

pg_restore: creating COMMENT "SCHEMA "public""

pg_restore: creating SCHEMA "qgis"

pg_restore: creating SCHEMA "regante"

pg_restore: creating SCHEMA "sde"

pg_restore: creating SCHEMA "sgp"

pg_restore: creating COMMENT "SCHEMA "sgp""

pg_restore: creating SCHEMA "sice"

pg_restore: creating SCHEMA "sig_owner"

pg_restore: creating SCHEMA "sig_owner2"

pg_restore: creating SCHEMA "simarsul"

pg_restore: creating SCHEMA "sisap"

pg_restore: creating SCHEMA "smg"

pg_restore: creating COMMENT "SCHEMA "smg""

pg_restore: creating SCHEMA "telegestao"

pg_restore: creating SCHEMA "tiger"

pg_restore: creating SCHEMA "tiger_data"

pg_restore: creating SCHEMA "topology"

pg_restore: creating EXTENSION "address_standardizer"

pg_restore: creating COMMENT "EXTENSION "address_standardizer""

pg_restore: creating EXTENSION "file_fdw"

pg_restore: creating COMMENT "EXTENSION "file_fdw""

pg_restore: creating EXTENSION "fuzzystrmatch"

pg_restore: creating COMMENT "EXTENSION "fuzzystrmatch""

pg_restore: creating EXTENSION "ogr_fdw"

pg_restore: creating COMMENT "EXTENSION "ogr_fdw""

pg_restore: creating EXTENSION "postgis"

pg_restore: creating COMMENT "EXTENSION "postgis""

pg_restore: creating EXTENSION "postgis_tiger_geocoder"

pg_restore: creating COMMENT "EXTENSION "postgis_tiger_geocoder""

pg_restore: creating EXTENSION "postgis_topology"

pg_restore: creating COMMENT "EXTENSION "postgis_topology""

pg_restore: creating EXTENSION "postgres_fdw"

pg_restore: creating COMMENT "EXTENSION "postgres_fdw""

pg_restore: creating EXTENSION "tablefunc"

pg_restore: creating COMMENT "EXTENSION "tablefunc""

pg_restore: creating TYPE "public.addbandarg"

pg_restore: creating COMMENT "public.TYPE "addbandarg""

pg_restore: creating TYPE "public.agg_count"

pg_restore: creating SHELL TYPE "public.raster"

pg_restore: creating FUNCTION "public.raster_in("cstring")"

pg_restore: creating FUNCTION "public.raster_out("public"."raster")"

pg_restore: creating TYPE "public.raster"

pg_restore: creating COMMENT "public.TYPE "raster""

pg_restore: creating TYPE "public.agg_samealignment"

pg_restore: creating SHELL TYPE "public.box2d"

pg_restore: creating FUNCTION "public.box2d_in("cstring")"

pg_restore: creating FUNCTION "public.box2d_out("public"."box2d")"

pg_restore: creating TYPE "public.box2d"

pg_restore: creating COMMENT "public.TYPE "box2d""

pg_restore: creating SHELL TYPE "public.box2df"

pg_restore: creating FUNCTION "public.box2df_in("cstring")"

pg_restore: creating FUNCTION "public.box2df_out("public"."box2df")"

pg_restore: creating TYPE "public.box2df"

pg_restore: creating SHELL TYPE "public.box3d"

pg_restore: creating FUNCTION "public.box3d_in("cstring")"

pg_restore: creating FUNCTION "public.box3d_out("public"."box3d")"

pg_restore: creating TYPE "public.box3d"

pg_restore: creating COMMENT "public.TYPE "box3d""

pg_restore: creating SHELL TYPE "public.geography"

pg_restore: creating FUNCTION "public.geography_analyze("internal")"

pg_restore: creating FUNCTION "public.geography_in("cstring", "oid", integer)"

pg_restore: creating FUNCTION "public.geography_out("public"."geography")"

pg_restore: creating FUNCTION "public.geography_recv("internal", "oid", integer)"

pg_restore: creating FUNCTION "public.geography_send("public"."geography")"

pg_restore: creating FUNCTION "public.geography_typmod_in("cstring"[])"

pg_restore: creating FUNCTION "public.geography_typmod_out(integer)"

pg_restore: creating TYPE "public.geography"

pg_restore: creating COMMENT "public.TYPE "geography""

pg_restore: creating SHELL TYPE "public.geometry"

pg_restore: creating FUNCTION "public.geometry_analyze("internal")"

pg_restore: creating FUNCTION "public.geometry_in("cstring")"

pg_restore: creating FUNCTION "public.geometry_out("public"."geometry")"

pg_restore: creating FUNCTION "public.geometry_recv("internal")"

pg_restore: creating FUNCTION "public.geometry_send("public"."geometry")"

pg_restore: creating FUNCTION "public.geometry_typmod_in("cstring"[])"

pg_restore: creating FUNCTION "public.geometry_typmod_out(integer)"

pg_restore: creating TYPE "public.geometry"

pg_restore: creating COMMENT "public.TYPE "geometry""

pg_restore: creating TYPE "public.geometry_dump"

pg_restore: creating COMMENT "public.TYPE "geometry_dump""

pg_restore: creating TYPE "public.geomval"

pg_restore: creating COMMENT "public.TYPE "geomval""

pg_restore: creating SHELL TYPE "public.gidx"

pg_restore: creating FUNCTION "public.gidx_in("cstring")"

pg_restore: creating FUNCTION "public.gidx_out("public"."gidx")"

pg_restore: creating TYPE "public.gidx"

pg_restore: creating SHELL TYPE "public.pgis_abs"

pg_restore: creating FUNCTION "public.pgis_abs_in("cstring")"

pg_restore: creating FUNCTION "public.pgis_abs_out("public"."pgis_abs")"

pg_restore: creating TYPE "public.pgis_abs"

pg_restore: creating TYPE "public.rastbandarg"

pg_restore: creating COMMENT "public.TYPE "rastbandarg""

pg_restore: creating TYPE "public.reclassarg"

pg_restore: creating COMMENT "public.TYPE "reclassarg""

pg_restore: creating SHELL TYPE "public.spheroid"

pg_restore: creating FUNCTION "public.spheroid_in("cstring")"

pg_restore: creating FUNCTION "public.spheroid_out("public"."spheroid")"

pg_restore: creating TYPE "public.spheroid"

pg_restore: creating TYPE "public.summarystats"

pg_restore: creating COMMENT "public.TYPE "summarystats""

pg_restore: creating TYPE "public.tablefunc_crosstab_2"

pg_restore: creating TYPE "public.tablefunc_crosstab_3"

pg_restore: creating TYPE "public.tablefunc_crosstab_4"

pg_restore: creating TYPE "public.unionarg"

pg_restore: creating COMMENT "public.TYPE "unionarg""

pg_restore: creating TYPE "public.valid_detail"

pg_restore: creating SHELL TYPE "sde.se_coord"

pg_restore: creating FUNCTION "sde.st_coord_in("cstring")"

pg_restore: creating FUNCTION "sde.st_coord_out("sde"."se_coord")"

pg_restore: creating FUNCTION "sde.st_coord_recv("internal")"

pg_restore: creating FUNCTION "sde.st_coord_send("sde"."se_coord")"

pg_restore: creating TYPE "sde.se_coord"

pg_restore: creating SHELL TYPE "sde.se_extent"

pg_restore: creating FUNCTION "sde.st_extent_in("cstring")"

pg_restore: creating FUNCTION "sde.st_extent_out("sde"."se_extent")"

pg_restore: creating FUNCTION "sde.st_extent_recv("internal")"

pg_restore: creating FUNCTION "sde.st_extent_send("sde"."se_extent")"

pg_restore: creating TYPE "sde.se_extent"

pg_restore: creating FUNCTION "sde.st_envelope_in("cstring")"

pg_restore: [archiver (db)] Error while PROCESSING TOC:

pg_restore: [archiver (db)] Error from TOC entry 4712; 1255 1141726 FUNCTION st_envelope_in("cstring") sde

pg_restore: [archiver (db)] could not execute query: ERROR:  duplicate key value violates unique constraint "pg_type_oid_index"

DETAIL:  Key (oid)=(1142573) already exists.

    Command was: CREATE FUNCTION "sde"."st_envelope_in"("cstring") RETURNS "sde"."st_envelope"

    LANGUAGE "c" IMMUTABLE STRICT

    AS 'st_g...

command: "D:\Program Files\PostgreSQL\9.5\bin/pg_restore" --port 50432 --username ^"postgres^" --exit-on-error --verbose --dbname ^"dbname^=postgis^" "pg_upgrade_dump_16393.custom" >> "pg_upgrade_dump_16393.log" 2>&1

 

 

 



Duarte Carreira
Diretor | Dep. Informação Geográfica e Cartografia
Tel. +351 284315100



Re: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

From
Tom Lane
Date:
Duarte Carreira <DCarreira@edia.pt> writes:
> I've trying to upgrade a 9.3 instance to 9.5 using pg_upgrade and facing this issue...

9.5.what?

Perusing the commit logs, I note that 9.5.3 included a fix for a
pg_upgrade issue that could possibly lead to this symptom, see
https://git.postgresql.org/gitweb/?p=postgresql.git&a=commitdiff&h=196870f2f

If you are on 9.5.recent, it'd be worth looking closer, because this
is certainly pretty odd.  One wouldn't expect a CREATE FUNCTION to
result in assignment of a type OID, at least not in pg_dump/pg_upgrade
scripts --- they should always put out a shell CREATE TYPE first.

By any chance, if you attempt a "pg_dump -s" from the problematic database,
does it emit any warnings (about dependency loops, perhaps)?

            regards, tom lane


Re: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

From
Duarte Carreira
Date:
Hi Tom.

It's 9.5 latest.

I'll try pgdump tomorrow.


From: Tom Lane <tgl@sss.pgh.pa.us>
Sent: Thursday, January 24, 2019 9:02:25 PM
To: Duarte Carreira
Cc: pgsql-general@postgresql.org
Subject: Re: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5
 
Duarte Carreira <DCarreira@edia.pt> writes:
> I've trying to upgrade a 9.3 instance to 9.5 using pg_upgrade and facing this issue...

9.5.what?

Perusing the commit logs, I note that 9.5.3 included a fix for a
pg_upgrade issue that could possibly lead to this symptom, see
https://git.postgresql.org/gitweb/?p=postgresql.git&a=commitdiff&h=196870f2f

If you are on 9.5.recent, it'd be worth looking closer, because this
is certainly pretty odd.  One wouldn't expect a CREATE FUNCTION to
result in assignment of a type OID, at least not in pg_dump/pg_upgrade
scripts --- they should always put out a shell CREATE TYPE first.

By any chance, if you attempt a "pg_dump -s" from the problematic database,
does it emit any warnings (about dependency loops, perhaps)?

                        regards, tom lane

RE: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

From
Duarte Carreira
Date:

Tom, just did a pg_dump -s from 9.3 and there are no warnings in the output file. It goes smoothly and fast (few seconds). Database is around 50GB, schema 9MB. Has postgis and a few more extensions – ogr_fdw (from postigs), file_fdw, postgres_fdw, table_func.

I also did a pg_dumpall -s and also no warnings.

 

I do a pg_dump full backup every week with no incidents.

 

Versions are:

 

Windows x64

9.3.22

9.5.15

 

Thanks for taking a look.

Duarte

 

De: Duarte Carreira
Enviada: 24 de janeiro de 2019 21:52
Para: Tom Lane <tgl@sss.pgh.pa.us>
Cc: pgsql-general@postgresql.org
Assunto: Re: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

 

Hi Tom.

It's 9.5 latest.

I'll try pgdump tomorrow.


From: Tom Lane <tgl@sss.pgh.pa.us>
Sent: Thursday, January 24, 2019 9:02:25 PM
To: Duarte Carreira
Cc: pgsql-general@postgresql.org
Subject: Re: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

 

Duarte Carreira <DCarreira@edia.pt> writes:
> I've trying to upgrade a 9.3 instance to 9.5 using pg_upgrade and facing this issue...

9.5.what?

Perusing the commit logs, I note that 9.5.3 included a fix for a
pg_upgrade issue that could possibly lead to this symptom, see
https://git.postgresql.org/gitweb/?p=postgresql.git&a=commitdiff&h=196870f2f

If you are on 9.5.recent, it'd be worth looking closer, because this
is certainly pretty odd.  One wouldn't expect a CREATE FUNCTION to
result in assignment of a type OID, at least not in pg_dump/pg_upgrade
scripts --- they should always put out a shell CREATE TYPE first.

By any chance, if you attempt a "pg_dump -s" from the problematic database,
does it emit any warnings (about dependency loops, perhaps)?

                        regards, tom lane

Re: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

From
Tom Lane
Date:
Duarte Carreira <DCarreira@edia.pt> writes:
> Tom, just did a pg_dump -s from 9.3 and there are no warnings in the output file. It goes smoothly and fast (few
seconds).Database is around 50GB, schema 9MB. Has postgis and a few more extensions - ogr_fdw (from postigs), file_fdw,
postgres_fdw,table_func. 

Hmph.  Would you be willing to send me a schema-only dump (off-list!)
of the problematic database?  The best format would be an "-Fc -s"
pg_dump output, because that would show what pg_dump thinks the
dependencies are.  Be sure you make it with the newer pg_dump.

            regards, tom lane


RE: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

From
Duarte Carreira
Date:
Sure, I'll be sending you the dump shortly off-list.

Duarte

-----Mensagem original-----
De: Tom Lane <tgl@sss.pgh.pa.us>
Enviada: 25 de janeiro de 2019 14:28
Para: Duarte Carreira <DCarreira@edia.pt>
Cc: pgsql-general@postgresql.org
Assunto: Re: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

Duarte Carreira <DCarreira@edia.pt> writes:
> Tom, just did a pg_dump -s from 9.3 and there are no warnings in the output file. It goes smoothly and fast (few
seconds).Database is around 50GB, schema 9MB. Has postgis and a few more extensions - ogr_fdw (from postigs), file_fdw,
postgres_fdw,table_func. 

Hmph.  Would you be willing to send me a schema-only dump (off-list!) of the problematic database?  The best format
wouldbe an "-Fc -s" 
pg_dump output, because that would show what pg_dump thinks the dependencies are.  Be sure you make it with the newer
pg_dump.

            regards, tom lane


RE: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

From
Duarte Carreira
Date:
Tom,

Used this command to dump the file attached:
"D:\Program Files\PostgreSQL\9.5\bin\pg_dump.exe" -Fc -p 5432 -U postgres -W -s postgis >dump_schema_db_postgis.custom

Hope you're able to pinpoint the issue.

Thanks again.
Duarte

-----Mensagem original-----
De: Duarte Carreira
Enviada: 25 de janeiro de 2019 15:01
Para: 'Tom Lane' <tgl@sss.pgh.pa.us>
Cc: pgsql-general@postgresql.org
Assunto: RE: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

Sure, I'll be sending you the dump shortly off-list.

Duarte

-----Mensagem original-----
De: Tom Lane <tgl@sss.pgh.pa.us>
Enviada: 25 de janeiro de 2019 14:28
Para: Duarte Carreira <DCarreira@edia.pt>
Cc: pgsql-general@postgresql.org
Assunto: Re: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

Duarte Carreira <DCarreira@edia.pt> writes:
> Tom, just did a pg_dump -s from 9.3 and there are no warnings in the output file. It goes smoothly and fast (few
seconds).Database is around 50GB, schema 9MB. Has postgis and a few more extensions - ogr_fdw (from postigs), file_fdw,
postgres_fdw,table_func. 

Hmph.  Would you be willing to send me a schema-only dump (off-list!) of the problematic database?  The best format
wouldbe an "-Fc -s" 
pg_dump output, because that would show what pg_dump thinks the dependencies are.  Be sure you make it with the newer
pg_dump.

            regards, tom lane

Attachment

Re: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

From
Tom Lane
Date:
Duarte Carreira <DCarreira@edia.pt> writes:
> Sure, I'll be sending you the dump shortly off-list.

Hmph.  So the problem seems to be that pg_dump isn't emitting a "shell
type" command for sde.st_envelope.  The first mention of that type is
in the "CREATE FUNCTION sde.st_envelope_in..." command, and that won't
have done anything to force the correct type OID to be assigned, and
indeed what you get happens to conflict with some other type.

Whatever the problem is only affects that one type --- if you grep
the pg_dump output for CREATE TYPE, you see

CREATE TYPE sde.se_coord;
CREATE TYPE sde.se_coord (
CREATE TYPE sde.se_extent;
CREATE TYPE sde.se_extent (
CREATE TYPE sde.st_envelope (
CREATE TYPE sde.st_geometry;
CREATE TYPE sde.st_geometry (
CREATE TYPE sde.st_pixeldata;
CREATE TYPE sde.st_pixeldata (
CREATE TYPE sde.st_raster;
CREATE TYPE sde.st_raster (
CREATE TYPE sde.st_state_data_type;
CREATE TYPE sde.st_state_data_type (

The lines with just a type name and no parameter list are the shell
type creation commands, and there's one for each base type ...
except st_envelope.

I'm not very sure what's going on here, but the logic that pg_dump
uses to decide whether to print a shell type assumes that there will
be circular dependencies between the base type and its I/O functions.
I wonder whether pg_depend has gotten damaged in your source database.

One thing you could check to start with is to manually try pg_dump's
dependency-fetch query in the problematic database:

explain SELECT classid, objid, refclassid, refobjid, deptype
FROM pg_depend WHERE deptype != 'p' AND deptype != 'e' ORDER BY 1,2;

If you get an indexscan plan, then it's possible that the issue
is a corrupted index, and "REINDEX pg_depend" would fix it.

But I suspect that it's going to be a seqscan-and-sort, which would mean
that the indexes couldn't be at fault and there actually is missing data
in pg_depend.  (The 9.3 release series had some messy data-loss problems,
so this conclusion isn't as astonishing as one could wish.)

The next thing to look at would be the dependencies associated with
the st_envelope type.  Try queries like this:

regression=# select pg_describe_object(refclassid,refobjid,refobjsubid), deptype from pg_depend where objid =
'public.widget'::regtype;
     pg_describe_object      | deptype
-----------------------------+---------
 schema public               | n
 function widget_in(cstring) | n
 function widget_out(widget) | n
(3 rows)

regression=# select pg_describe_object(classid,objid,objsubid), deptype from pg_depend where refobjid =
'public.widget'::regtype;
         pg_describe_object          | deptype
-------------------------------------+---------
 function widget_in(cstring)         | n
 function widget_out(widget)         | n
 type widget[]                       | i
 ... more stuff ...

I don't have postgis installed here, so this example is looking at
a type "public.widget", but of course what you want to look at is
sde.st_envelope.  If you don't see links to st_envelope_in and
st_envelope_out in both queries, then we've found the problem.

Fixing it is a bit trickier, but in principle you could manually
insert the missing row(s) once you know what they need to be.

            regards, tom lane


RE: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

From
Duarte Carreira
Date:
First, I'd like to apologize for posting the dump file to the whole list... I wish I could delete it, at least the
attachment...

I ran the explain plan, and got a seq scan, so no faulty index:
"Sort  (cost=5073.45..5195.00 rows=48622 width=17)"
"  Sort Key: classid, objid"
"  ->  Seq Scan on pg_depend  (cost=0.00..1288.39 rows=48622 width=17)"
"        Filter: ((deptype <> 'p'::"char") AND (deptype <> 'e'::"char"))"

So I proceeded with the 2 queries:

select pg_describe_object(refclassid,refobjid,refobjsubid), deptype from pg_depend where objid =
'sde.st_envelope'::regtype;
"schema sde";"n"
"function st_envelope_in(cstring)";"n"
"function st_envelope_out(st_envelope)";"n"

select pg_describe_object(classid,objid,objsubid), deptype from pg_depend where refobjid = 'sde.st_envelope'::regtype;
"type st_envelope[]";"i"
"function st_envelope_union(st_envelope,st_envelope)";"n"
"function st_envelope_union(st_envelope,st_envelope)";"n"
"function st_envelope_union(st_envelope,st_envelope)";"n"
"function st_envelope_intersects(st_envelope,st_envelope)";"n"
"function st_envelope_intersects(st_envelope,st_envelope)";"n"
"function st_envelope_intersects(st_envelope,st_envelope)";"n"
"function st_envelope_equal(st_envelope,st_envelope)";"n"
"function st_envelope_equal(st_envelope,st_envelope)";"n"
"function st_geo_gist_equal(st_envelope,st_envelope,internal)";"n"
"function st_geo_gist_equal(st_envelope,st_envelope,internal)";"n"

But I couldn't figure out what this means...

Duarte

-----Mensagem original-----
De: Tom Lane <tgl@sss.pgh.pa.us>
Enviada: 25 de janeiro de 2019 16:04
Para: Duarte Carreira <DCarreira@edia.pt>
Cc: pgsql-general@postgresql.org
Assunto: Re: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

Duarte Carreira <DCarreira@edia.pt> writes:
> Sure, I'll be sending you the dump shortly off-list.

Hmph.  So the problem seems to be that pg_dump isn't emitting a "shell type" command for sde.st_envelope.  The first
mentionof that type is in the "CREATE FUNCTION sde.st_envelope_in..." command, and that won't have done anything to
forcethe correct type OID to be assigned, and indeed what you get happens to conflict with some other type. 

Whatever the problem is only affects that one type --- if you grep the pg_dump output for CREATE TYPE, you see

CREATE TYPE sde.se_coord;
CREATE TYPE sde.se_coord (
CREATE TYPE sde.se_extent;
CREATE TYPE sde.se_extent (
CREATE TYPE sde.st_envelope (
CREATE TYPE sde.st_geometry;
CREATE TYPE sde.st_geometry (
CREATE TYPE sde.st_pixeldata;
CREATE TYPE sde.st_pixeldata (
CREATE TYPE sde.st_raster;
CREATE TYPE sde.st_raster (
CREATE TYPE sde.st_state_data_type;
CREATE TYPE sde.st_state_data_type (

The lines with just a type name and no parameter list are the shell type creation commands, and there's one for each
basetype ... 
except st_envelope.

I'm not very sure what's going on here, but the logic that pg_dump uses to decide whether to print a shell type assumes
thatthere will be circular dependencies between the base type and its I/O functions. 
I wonder whether pg_depend has gotten damaged in your source database.

One thing you could check to start with is to manually try pg_dump's dependency-fetch query in the problematic
database:

explain SELECT classid, objid, refclassid, refobjid, deptype FROM pg_depend WHERE deptype != 'p' AND deptype != 'e'
ORDERBY 1,2; 

If you get an indexscan plan, then it's possible that the issue is a corrupted index, and "REINDEX pg_depend" would fix
it.

But I suspect that it's going to be a seqscan-and-sort, which would mean that the indexes couldn't be at fault and
thereactually is missing data in pg_depend.  (The 9.3 release series had some messy data-loss problems, so this
conclusionisn't as astonishing as one could wish.) 

The next thing to look at would be the dependencies associated with the st_envelope type.  Try queries like this:

regression=# select pg_describe_object(refclassid,refobjid,refobjsubid), deptype from pg_depend where objid =
'public.widget'::regtype;
     pg_describe_object      | deptype
-----------------------------+---------
 schema public               | n
 function widget_in(cstring) | n
 function widget_out(widget) | n
(3 rows)

regression=# select pg_describe_object(classid,objid,objsubid), deptype from pg_depend where refobjid =
'public.widget'::regtype;
         pg_describe_object          | deptype
-------------------------------------+---------
 function widget_in(cstring)         | n
 function widget_out(widget)         | n
 type widget[]                       | i
 ... more stuff ...

I don't have postgis installed here, so this example is looking at a type "public.widget", but of course what you want
tolook at is sde.st_envelope.  If you don't see links to st_envelope_in and st_envelope_out in both queries, then we've
foundthe problem. 

Fixing it is a bit trickier, but in principle you could manually insert the missing row(s) once you know what they need
tobe. 

            regards, tom lane


RE: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

From
Duarte Carreira
Date:
Just a footnote. This is not a postgis object, it's an esri object. Belongs to the sde schema that is installed in
everydatabase where esri software is installed in, namely arcgis server. 

-----Mensagem original-----
De: Duarte Carreira
Enviada: 25 de janeiro de 2019 18:03
Para: 'Tom Lane' <tgl@sss.pgh.pa.us>
Cc: pgsql-general@postgresql.org
Assunto: RE: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

First, I'd like to apologize for posting the dump file to the whole list... I wish I could delete it, at least the
attachment...

I ran the explain plan, and got a seq scan, so no faulty index:
"Sort  (cost=5073.45..5195.00 rows=48622 width=17)"
"  Sort Key: classid, objid"
"  ->  Seq Scan on pg_depend  (cost=0.00..1288.39 rows=48622 width=17)"
"        Filter: ((deptype <> 'p'::"char") AND (deptype <> 'e'::"char"))"

So I proceeded with the 2 queries:

select pg_describe_object(refclassid,refobjid,refobjsubid), deptype from pg_depend where objid =
'sde.st_envelope'::regtype;"schema sde";"n" 
"function st_envelope_in(cstring)";"n"
"function st_envelope_out(st_envelope)";"n"

select pg_describe_object(classid,objid,objsubid), deptype from pg_depend where refobjid = 'sde.st_envelope'::regtype;
"typest_envelope[]";"i" 
"function st_envelope_union(st_envelope,st_envelope)";"n"
"function st_envelope_union(st_envelope,st_envelope)";"n"
"function st_envelope_union(st_envelope,st_envelope)";"n"
"function st_envelope_intersects(st_envelope,st_envelope)";"n"
"function st_envelope_intersects(st_envelope,st_envelope)";"n"
"function st_envelope_intersects(st_envelope,st_envelope)";"n"
"function st_envelope_equal(st_envelope,st_envelope)";"n"
"function st_envelope_equal(st_envelope,st_envelope)";"n"
"function st_geo_gist_equal(st_envelope,st_envelope,internal)";"n"
"function st_geo_gist_equal(st_envelope,st_envelope,internal)";"n"

But I couldn't figure out what this means...

Duarte

-----Mensagem original-----
De: Tom Lane <tgl@sss.pgh.pa.us>
Enviada: 25 de janeiro de 2019 16:04
Para: Duarte Carreira <DCarreira@edia.pt>
Cc: pgsql-general@postgresql.org
Assunto: Re: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

Duarte Carreira <DCarreira@edia.pt> writes:
> Sure, I'll be sending you the dump shortly off-list.

Hmph.  So the problem seems to be that pg_dump isn't emitting a "shell type" command for sde.st_envelope.  The first
mentionof that type is in the "CREATE FUNCTION sde.st_envelope_in..." command, and that won't have done anything to
forcethe correct type OID to be assigned, and indeed what you get happens to conflict with some other type. 

Whatever the problem is only affects that one type --- if you grep the pg_dump output for CREATE TYPE, you see

CREATE TYPE sde.se_coord;
CREATE TYPE sde.se_coord (
CREATE TYPE sde.se_extent;
CREATE TYPE sde.se_extent (
CREATE TYPE sde.st_envelope (
CREATE TYPE sde.st_geometry;
CREATE TYPE sde.st_geometry (
CREATE TYPE sde.st_pixeldata;
CREATE TYPE sde.st_pixeldata (
CREATE TYPE sde.st_raster;
CREATE TYPE sde.st_raster (
CREATE TYPE sde.st_state_data_type;
CREATE TYPE sde.st_state_data_type (

The lines with just a type name and no parameter list are the shell type creation commands, and there's one for each
basetype ... 
except st_envelope.

I'm not very sure what's going on here, but the logic that pg_dump uses to decide whether to print a shell type assumes
thatthere will be circular dependencies between the base type and its I/O functions. 
I wonder whether pg_depend has gotten damaged in your source database.

One thing you could check to start with is to manually try pg_dump's dependency-fetch query in the problematic
database:

explain SELECT classid, objid, refclassid, refobjid, deptype FROM pg_depend WHERE deptype != 'p' AND deptype != 'e'
ORDERBY 1,2; 

If you get an indexscan plan, then it's possible that the issue is a corrupted index, and "REINDEX pg_depend" would fix
it.

But I suspect that it's going to be a seqscan-and-sort, which would mean that the indexes couldn't be at fault and
thereactually is missing data in pg_depend.  (The 9.3 release series had some messy data-loss problems, so this
conclusionisn't as astonishing as one could wish.) 

The next thing to look at would be the dependencies associated with the st_envelope type.  Try queries like this:

regression=# select pg_describe_object(refclassid,refobjid,refobjsubid), deptype from pg_depend where objid =
'public.widget'::regtype;
     pg_describe_object      | deptype
-----------------------------+---------
 schema public               | n
 function widget_in(cstring) | n
 function widget_out(widget) | n
(3 rows)

regression=# select pg_describe_object(classid,objid,objsubid), deptype from pg_depend where refobjid =
'public.widget'::regtype;
         pg_describe_object          | deptype
-------------------------------------+---------
 function widget_in(cstring)         | n
 function widget_out(widget)         | n
 type widget[]                       | i
 ... more stuff ...

I don't have postgis installed here, so this example is looking at a type "public.widget", but of course what you want
tolook at is sde.st_envelope.  If you don't see links to st_envelope_in and st_envelope_out in both queries, then we've
foundthe problem. 

Fixing it is a bit trickier, but in principle you could manually insert the missing row(s) once you know what they need
tobe. 

            regards, tom lane


Re: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

From
Tom Lane
Date:
Duarte Carreira <DCarreira@edia.pt> writes:
> So I proceeded with the 2 queries:

> select pg_describe_object(refclassid,refobjid,refobjsubid), deptype from pg_depend where objid =
'sde.st_envelope'::regtype;
> "schema sde";"n"
> "function st_envelope_in(cstring)";"n"
> "function st_envelope_out(st_envelope)";"n"

OK, that looks about like what I'd expect.

> select pg_describe_object(classid,objid,objsubid), deptype from pg_depend where refobjid =
'sde.st_envelope'::regtype;
> "type st_envelope[]";"i"
> "function st_envelope_union(st_envelope,st_envelope)";"n"
> "function st_envelope_union(st_envelope,st_envelope)";"n"
> "function st_envelope_union(st_envelope,st_envelope)";"n"
> "function st_envelope_intersects(st_envelope,st_envelope)";"n"
> "function st_envelope_intersects(st_envelope,st_envelope)";"n"
> "function st_envelope_intersects(st_envelope,st_envelope)";"n"
> "function st_envelope_equal(st_envelope,st_envelope)";"n"
> "function st_envelope_equal(st_envelope,st_envelope)";"n"
> "function st_geo_gist_equal(st_envelope,st_envelope,internal)";"n"
> "function st_geo_gist_equal(st_envelope,st_envelope,internal)";"n"

Ah-hah --- st_envelope_in and st_envelope_out are not mentioned here?
That explains your problem.  You'd need to add those two rows to pg_depend,
which could go something like

insert into pg_depend (classid, objid, objsubid,
                       refclassid, refobjid, refobjsubid, deptype)
values (
  'pg_proc'::regclass,
  'sde.st_envelope_in(cstring)'::regprocedure,
  0,
  'pg_type'::regclass,
  'sde.st_envelope'::regtype,
  0,
  'n');

insert into pg_depend (classid, objid, objsubid,
                       refclassid, refobjid, refobjsubid, deptype)
values (
  'pg_proc'::regclass,
  'sde.st_envelope_out(sde.st_envelope)'::regprocedure,
  0,
  'pg_type'::regclass,
  'sde.st_envelope'::regtype,
  0,
  'n');

I suppose the evidence about what happened to those rows is long gone,
so there's not much point in doing anything but patching things up to
the point where you can run pg_upgrade.

            regards, tom lane


RE: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

From
Duarte Carreira
Date:
Tom, can't thank you enough.

Now, the 2nd query and results are like so:
select pg_describe_object(classid,objid,objsubid), deptype from pg_depend where refobjid = 'sde.st_envelope'::regtype;
"function st_envelope_in(cstring)";"n"
"function st_envelope_out(st_envelope)";"n"
"type st_envelope[]";"i"
"function st_envelope_union(st_envelope,st_envelope)";"n"
"function st_envelope_union(st_envelope,st_envelope)";"n"
"function st_envelope_union(st_envelope,st_envelope)";"n"
"function st_envelope_intersects(st_envelope,st_envelope)";"n"
"function st_envelope_intersects(st_envelope,st_envelope)";"n"
"function st_envelope_intersects(st_envelope,st_envelope)";"n"
"function st_envelope_equal(st_envelope,st_envelope)";"n"
"function st_envelope_equal(st_envelope,st_envelope)";"n"
"function st_geo_gist_equal(st_envelope,st_envelope,internal)";"n"
"function st_geo_gist_equal(st_envelope,st_envelope,internal)";"n"

I'll try pg_upgrade again and let you know how it goes. (fingers crossed)

Thanks,
Duarte

-----Mensagem original-----
De: Tom Lane <tgl@sss.pgh.pa.us>
Enviada: 25 de janeiro de 2019 18:19
Para: Duarte Carreira <DCarreira@edia.pt>
Cc: pgsql-general@postgresql.org
Assunto: Re: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

Duarte Carreira <DCarreira@edia.pt> writes:
> So I proceeded with the 2 queries:

> select pg_describe_object(refclassid,refobjid,refobjsubid), deptype
> from pg_depend where objid = 'sde.st_envelope'::regtype; "schema sde";"n"
> "function st_envelope_in(cstring)";"n"
> "function st_envelope_out(st_envelope)";"n"

OK, that looks about like what I'd expect.

> select pg_describe_object(classid,objid,objsubid), deptype from
> pg_depend where refobjid = 'sde.st_envelope'::regtype; "type st_envelope[]";"i"
> "function st_envelope_union(st_envelope,st_envelope)";"n"
> "function st_envelope_union(st_envelope,st_envelope)";"n"
> "function st_envelope_union(st_envelope,st_envelope)";"n"
> "function st_envelope_intersects(st_envelope,st_envelope)";"n"
> "function st_envelope_intersects(st_envelope,st_envelope)";"n"
> "function st_envelope_intersects(st_envelope,st_envelope)";"n"
> "function st_envelope_equal(st_envelope,st_envelope)";"n"
> "function st_envelope_equal(st_envelope,st_envelope)";"n"
> "function st_geo_gist_equal(st_envelope,st_envelope,internal)";"n"
> "function st_geo_gist_equal(st_envelope,st_envelope,internal)";"n"

Ah-hah --- st_envelope_in and st_envelope_out are not mentioned here?
That explains your problem.  You'd need to add those two rows to pg_depend, which could go something like

insert into pg_depend (classid, objid, objsubid,
                       refclassid, refobjid, refobjsubid, deptype) values (
  'pg_proc'::regclass,
  'sde.st_envelope_in(cstring)'::regprocedure,
  0,
  'pg_type'::regclass,
  'sde.st_envelope'::regtype,
  0,
  'n');

insert into pg_depend (classid, objid, objsubid,
                       refclassid, refobjid, refobjsubid, deptype) values (
  'pg_proc'::regclass,
  'sde.st_envelope_out(sde.st_envelope)'::regprocedure,
  0,
  'pg_type'::regclass,
  'sde.st_envelope'::regtype,
  0,
  'n');

I suppose the evidence about what happened to those rows is long gone, so there's not much point in doing anything but
patchingthings up to the point where you can run pg_upgrade. 

            regards, tom lane


RE: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

From
Duarte Carreira
Date:
Well, happy to report pg_upgrade worked, and have 9.5 cluster up and running.

Will be testing during weekend but everything looks ok. Just had to tweak a few settings that changed from 9.3 to 9.5
(logsand checkpoint_segments). 

Life's good.

Thanks again, and just have to say it - open source rocks!

Duarte

-----Mensagem original-----
De: Duarte Carreira
Enviada: 25 de janeiro de 2019 18:24
Para: Tom Lane <tgl@sss.pgh.pa.us>
Cc: pgsql-general@postgresql.org
Assunto: RE: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

Tom, can't thank you enough.

Now, the 2nd query and results are like so:
select pg_describe_object(classid,objid,objsubid), deptype from pg_depend where refobjid = 'sde.st_envelope'::regtype;
"functionst_envelope_in(cstring)";"n" 
"function st_envelope_out(st_envelope)";"n"
"type st_envelope[]";"i"
"function st_envelope_union(st_envelope,st_envelope)";"n"
"function st_envelope_union(st_envelope,st_envelope)";"n"
"function st_envelope_union(st_envelope,st_envelope)";"n"
"function st_envelope_intersects(st_envelope,st_envelope)";"n"
"function st_envelope_intersects(st_envelope,st_envelope)";"n"
"function st_envelope_intersects(st_envelope,st_envelope)";"n"
"function st_envelope_equal(st_envelope,st_envelope)";"n"
"function st_envelope_equal(st_envelope,st_envelope)";"n"
"function st_geo_gist_equal(st_envelope,st_envelope,internal)";"n"
"function st_geo_gist_equal(st_envelope,st_envelope,internal)";"n"

I'll try pg_upgrade again and let you know how it goes. (fingers crossed)

Thanks,
Duarte

-----Mensagem original-----
De: Tom Lane <tgl@sss.pgh.pa.us>
Enviada: 25 de janeiro de 2019 18:19
Para: Duarte Carreira <DCarreira@edia.pt>
Cc: pgsql-general@postgresql.org
Assunto: Re: duplicate OID issue when using pg_upgrade to move from 9.3 to 9.5

Duarte Carreira <DCarreira@edia.pt> writes:
> So I proceeded with the 2 queries:

> select pg_describe_object(refclassid,refobjid,refobjsubid), deptype
> from pg_depend where objid = 'sde.st_envelope'::regtype; "schema sde";"n"
> "function st_envelope_in(cstring)";"n"
> "function st_envelope_out(st_envelope)";"n"

OK, that looks about like what I'd expect.

> select pg_describe_object(classid,objid,objsubid), deptype from
> pg_depend where refobjid = 'sde.st_envelope'::regtype; "type st_envelope[]";"i"
> "function st_envelope_union(st_envelope,st_envelope)";"n"
> "function st_envelope_union(st_envelope,st_envelope)";"n"
> "function st_envelope_union(st_envelope,st_envelope)";"n"
> "function st_envelope_intersects(st_envelope,st_envelope)";"n"
> "function st_envelope_intersects(st_envelope,st_envelope)";"n"
> "function st_envelope_intersects(st_envelope,st_envelope)";"n"
> "function st_envelope_equal(st_envelope,st_envelope)";"n"
> "function st_envelope_equal(st_envelope,st_envelope)";"n"
> "function st_geo_gist_equal(st_envelope,st_envelope,internal)";"n"
> "function st_geo_gist_equal(st_envelope,st_envelope,internal)";"n"

Ah-hah --- st_envelope_in and st_envelope_out are not mentioned here?
That explains your problem.  You'd need to add those two rows to pg_depend, which could go something like

insert into pg_depend (classid, objid, objsubid,
                       refclassid, refobjid, refobjsubid, deptype) values (
  'pg_proc'::regclass,
  'sde.st_envelope_in(cstring)'::regprocedure,
  0,
  'pg_type'::regclass,
  'sde.st_envelope'::regtype,
  0,
  'n');

insert into pg_depend (classid, objid, objsubid,
                       refclassid, refobjid, refobjsubid, deptype) values (
  'pg_proc'::regclass,
  'sde.st_envelope_out(sde.st_envelope)'::regprocedure,
  0,
  'pg_type'::regclass,
  'sde.st_envelope'::regtype,
  0,
  'n');

I suppose the evidence about what happened to those rows is long gone, so there's not much point in doing anything but
patchingthings up to the point where you can run pg_upgrade. 

            regards, tom lane