Thread: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
adrian.klaver@gmail.com
Date:
The following bug has been logged on the website:

Bug reference:      6666
Logged by:          Adrian Klaver
Email address:      adrian.klaver@gmail.com
PostgreSQL version: 9.0.7
Operating system:   Linux/OpenSuse 12.1
Description:=20=20=20=20=20=20=20=20

I thought this was fixed in:

http://git.postgresql.org/gitweb/?p=3Dpostgresql.git;a=3Dcommit;h=3Dcfe443a=
b9d42b4ffe950608f01c3a4bdc2895c7b

I just tried an upgrade from 9.0.7 to 9.2beta1 using pg_upgrade. I ran it
first with -c and everything passed. When I did a live upgrade it failed
with this message in pg_upgrade_restore.log:

CREATE FUNCTION plpython_call_handler() RETURNS language_handler
    LANGUAGE c
    AS '$libdir/plpython', 'plpython_call_handler';
psql:pg_upgrade_dump_db.sql:13541: ERROR:  could not access file
"$libdir/plpython": No such file or directory

That is indeed true, there is no plpython, there is plpython2 instead. I
created a symlink plpython --> plpython2 and the upgraded succeeded after a
re-initdb of the new cluster.=20

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Bruce Momjian
Date:
On Sat, May 26, 2012 at 11:42:09PM +0000, adrian.klaver@gmail.com wrote:
> The following bug has been logged on the website:
>
> Bug reference:      6666
> Logged by:          Adrian Klaver
> Email address:      adrian.klaver@gmail.com
> PostgreSQL version: 9.0.7
> Operating system:   Linux/OpenSuse 12.1
> Description:
>
> I thought this was fixed in:
>
> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=cfe443ab9d42b4ffe950608f01c3a4bdc2895c7b
>
> I just tried an upgrade from 9.0.7 to 9.2beta1 using pg_upgrade. I ran it
> first with -c and everything passed. When I did a live upgrade it failed
> with this message in pg_upgrade_restore.log:
>
> CREATE FUNCTION plpython_call_handler() RETURNS language_handler
>     LANGUAGE c
>     AS '$libdir/plpython', 'plpython_call_handler';
> psql:pg_upgrade_dump_db.sql:13541: ERROR:  could not access file
> "$libdir/plpython": No such file or directory
>
> That is indeed true, there is no plpython, there is plpython2 instead. I
> created a symlink plpython --> plpython2 and the upgraded succeeded after a
> re-initdb of the new cluster.

It took me a little while to remember the cause of this problem, but I
found it!

     http://archives.postgresql.org/pgsql-hackers/2012-03/msg01101.php

This was reported in March, 2012 --- please read the entire thread.  The
problem is that you have a plpython_call_handler defined in the "public"
schema, and it is being dumped out by pg_dumpall, rather than using a
generic CREATE LANGUAGE statement.  Odds are this is left over from an
older version of Postgres.

If you can help me find out how these got defined this way, I might be
able to prevent this problem for the next person.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Adrian Klaver
Date:
On 05/26/2012 07:24 PM, Bruce Momjian wrote:
> On Sat, May 26, 2012 at 11:42:09PM +0000, adrian.klaver@gmail.com wrote:
>> The following bug has been logged on the website:
>>
>> Bug reference:      6666
>> Logged by:          Adrian Klaver
>> Email address:      adrian.klaver@gmail.com
>> PostgreSQL version: 9.0.7
>> Operating system:   Linux/OpenSuse 12.1
>> Description:
>>

>>
>> I just tried an upgrade from 9.0.7 to 9.2beta1 using pg_upgrade. I ran it
>> first with -c and everything passed. When I did a live upgrade it failed
>> with this message in pg_upgrade_restore.log:
>>
>> CREATE FUNCTION plpython_call_handler() RETURNS language_handler
>>      LANGUAGE c
>>      AS '$libdir/plpython', 'plpython_call_handler';
>> psql:pg_upgrade_dump_db.sql:13541: ERROR:  could not access file
>> "$libdir/plpython": No such file or directory
>>
>> That is indeed true, there is no plpython, there is plpython2 instead. I
>> created a symlink plpython -->  plpython2 and the upgraded succeeded after a
>> re-initdb of the new cluster.
>
> It took me a little while to remember the cause of this problem, but I
> found it!
>
>      http://archives.postgresql.org/pgsql-hackers/2012-03/msg01101.php
>
> This was reported in March, 2012 --- please read the entire thread.  The
> problem is that you have a plpython_call_handler defined in the "public"
> schema, and it is being dumped out by pg_dumpall, rather than using a
> generic CREATE LANGUAGE statement.  Odds are this is left over from an
> older version of Postgres.

This particular database dates back to version 7.2, though plpythonu was not
added until version 8.0.x using the following:

CREATE PROCEDURAL LANGUAGE 'plpythonu'
  HANDLER plpython_call_handler;

Until now my procedure has been to do dump/restore to move from one major version to another.

>
> If you can help me find out how these got defined this way, I might be
> able to prevent this problem for the next person.
>

After reading the above thread here is what the queries mentioned return:

production=#  SELECT nspname,proname,probin FROM pg_proc,pg_namespace
WHERE probin LIKE '%python%' and pg_proc.pronamespace=pg_namespace.oid;

nspname   |         proname         |      probin
------------+-------------------------+------------------
 pg_catalog | plpython_call_handler   | $libdir/plpython
 pg_catalog | plpython_inline_handler | $libdir/plpython
 public     | plpython_call_handler   | $libdir/plpython
(3 rows)



production=# SELECT tableoid, oid, proname, prolang, pronargs, proargtypes,
prorettype, proacl, pronamespace, (SELECT rolname FROM pg_catalog.pg_roles WHERE
oid = proowner) AS rolname FROM pg_proc p WHERE NOT proisagg AND
(pronamespace != (SELECT oid FROM pg_namespace WHERE
nspname = 'pg_catalog'));
 tableoid | oid | proname | prolang | pronargs | proargtypes | prorettype | proacl | pronamespace | rolname
----------+-----+---------+---------+----------+-------------+------------+--------+--------------+---------
(0 rows)

If you need any more information let me know.

Thanks,

--
Adrian Klaver
adrian.klaver@gmail.com

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Tom Lane
Date:
Adrian Klaver <adrian.klaver@gmail.com> writes:
> After reading the above thread here is what the queries mentioned return:

> production=#  SELECT nspname,proname,probin FROM pg_proc,pg_namespace
> WHERE probin LIKE '%python%' and pg_proc.pronamespace=pg_namespace.oid;

> nspname   |         proname         |      probin
> ------------+-------------------------+------------------
>  pg_catalog | plpython_call_handler   | $libdir/plpython
>  pg_catalog | plpython_inline_handler | $libdir/plpython
>  public     | plpython_call_handler   | $libdir/plpython
> (3 rows)

I think what you need to do is drop the one in public, ie
    drop function public.plpython_call_handler();
The other two are what the language is actually using nowadays.

Hopefully pg_upgrade will then cope with upgrading them ...

            regards, tom lane

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Adrian Klaver
Date:
On 05/27/2012 01:35 PM, Tom Lane wrote:
> Adrian Klaver<adrian.klaver@gmail.com>  writes:
>> After reading the above thread here is what the queries mentioned return:
>
>> production=#  SELECT nspname,proname,probin FROM pg_proc,pg_namespace
>> WHERE probin LIKE '%python%' and pg_proc.pronamespace=pg_namespace.oid;
>
>> nspname   |         proname         |      probin
>> ------------+-------------------------+------------------
>>   pg_catalog | plpython_call_handler   | $libdir/plpython
>>   pg_catalog | plpython_inline_handler | $libdir/plpython
>>   public     | plpython_call_handler   | $libdir/plpython
>> (3 rows)
>
> I think what you need to do is drop the one in public, ie
>     drop function public.plpython_call_handler();
> The other two are what the language is actually using nowadays.

I will give that a try.

Though knowing what the problem is I wonder if another solution is to
make the plpythonu lib layout follow the description of the 2/3 split
detailed here:

http://www.postgresql.org/docs/9.2/static/plpython-python23.html

In particular:
"The language named plpythonu implements PL/Python based on the default
Python language variant, which is currently Python 2"

In other words automatically create a symlink:

plpython.so -> plpython2.so*

>
> Hopefully pg_upgrade will then cope with upgrading them ...
>
>             regards, tom lane


--
Adrian Klaver
adrian.klaver@gmail.com

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Adrian Klaver
Date:
On 05/27/2012 01:35 PM, Tom Lane wrote:
> Adrian Klaver<adrian.klaver@gmail.com>  writes:
>> After reading the above thread here is what the queries mentioned return:
>
>> production=#  SELECT nspname,proname,probin FROM pg_proc,pg_namespace
>> WHERE probin LIKE '%python%' and pg_proc.pronamespace=pg_namespace.oid;
>
>> nspname   |         proname         |      probin
>> ------------+-------------------------+------------------
>>   pg_catalog | plpython_call_handler   | $libdir/plpython
>>   pg_catalog | plpython_inline_handler | $libdir/plpython
>>   public     | plpython_call_handler   | $libdir/plpython
>> (3 rows)
>
> I think what you need to do is drop the one in public, ie
>     drop function public.plpython_call_handler();
> The other two are what the language is actually using nowadays.
>
> Hopefully pg_upgrade will then cope with upgrading them ...

To follow up, I dropped the public function and reran pg_upgrade from
scratch and it completed successfully.

>
>             regards, tom lane


--
Adrian Klaver
adrian.klaver@gmail.com

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Bruce Momjian
Date:
On Sun, May 27, 2012 at 11:49:31AM -0700, Adrian Klaver wrote:
> >
> > If you can help me find out how these got defined this way, I might be
> > able to prevent this problem for the next person.
> >
>
> After reading the above thread here is what the queries mentioned return:
>
> production=#  SELECT nspname,proname,probin FROM pg_proc,pg_namespace
> WHERE probin LIKE '%python%' and pg_proc.pronamespace=pg_namespace.oid;
>
> nspname   |         proname         |      probin
> ------------+-------------------------+------------------
>  pg_catalog | plpython_call_handler   | $libdir/plpython
>  pg_catalog | plpython_inline_handler | $libdir/plpython
>  public     | plpython_call_handler   | $libdir/plpython
> (3 rows)

You are the third person to have this problem, and this was the
information I needed to properly find the cause.  I believe it was done
in this 8.1 commit:

    e0dedd0559f005d60c69c9772163e69c204bac69

    Implement a preliminary 'template' facility for procedural languages,
    as per my recent proposal.  For now the template data is hard-wired
    in proclang.c --- this should be replaced later by a new shared
    system catalog, but we don't want to force initdb during 8.1 beta.
    This change lets us cleanly load existing dump files even if they
    contain outright wrong information about a PL's support functions, such
    as a wrong path to the shared library or a missing validator function.
    Also, we can revert the recent kluges to make pg_dump dump PL support
    functions that are stored in pg_catalog.  While at it, I removed the
    code in pg_regress that replaced $libdir with a hardcoded path for
    temporary installations.  This is no longer needed given our support
    for relocatable installations.

This moved the helper functions into pg_catalog, but the author probably
didn't realize that public schema helper functions would continue to be
dumped by pg_dump.  These helper functions continued to be
dumped/restored until the rename.  There are certainly helper functions
for other languages that are still duplicated in the public schema ---
there is nothing unique about plpython.  We are only seeing problems
because of the plpython.so rename.

In normal use, a pg_dumpall restore would throw an error about a missing
helper function shared objects, but the pg_catalog entry would continue
to work just fine.  Odd we have not heard complaints from users seeing
that error on restore --- odds are, they are just ignoring the error,
which pg_upgrade does not do.

The attached pg_upgrade patch adds checks for this plpython helper
function and reports a proper error, suggesting how to fix the problem:

    Performing Consistency Checks
    -----------------------------
    Checking current, bin, and data directories                 ok
    Checking cluster versions                                   ok

    The old cluster has a "plpython_call_handler" function defined
    in the "public" schema which is a duplicate of the one defined
    in the "pg_catalog" schema.  You can confirm this by executing
    in psql:

            \df *.plpython_call_handler

    The "public" schema version of this function was created by a
    pre-8.1 install of plpython, and must be removed for pg_upgrade
    to complete because it references a now-obsolete "plpython"
    shared object file.  You can remove the "public" schema version
    of this function by running the following command:

            DROP FUNCTION public.plpython_call_handler()

    in each affected database:

            test
            test3

    Remove the problem functions from the old cluster to continue.
    Failure, exiting

We could do the same for other PL languages if they are ever renamed.  I
suppose we don't care about fixing the duplicate schema entries.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Attachment

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Adrian Klaver
Date:
On 05/29/2012 05:44 PM, Bruce Momjian wrote:
> On Sun, May 27, 2012 at 11:49:31AM -0700, Adrian Klaver wrote:
>>>
>>> If you can help me find out how these got defined this way, I might be
>>> able to prevent this problem for the next person.
>>>
>>
>> After reading the above thread here is what the queries mentioned return:
>>
>> production=#  SELECT nspname,proname,probin FROM pg_proc,pg_namespace
>> WHERE probin LIKE '%python%' and pg_proc.pronamespace=pg_namespace.oid;
>>
>> nspname   |         proname         |      probin
>> ------------+-------------------------+------------------
>>   pg_catalog | plpython_call_handler   | $libdir/plpython
>>   pg_catalog | plpython_inline_handler | $libdir/plpython
>>   public     | plpython_call_handler   | $libdir/plpython
>> (3 rows)
>
> You are the third person to have this problem, and this was the
> information I needed to properly find the cause.  I believe it was done
> in this 8.1 commit:
>
>      e0dedd0559f005d60c69c9772163e69c204bac69
>
>      Implement a preliminary 'template' facility for procedural languages,
>      as per my recent proposal.  For now the template data is hard-wired
>      in proclang.c --- this should be replaced later by a new shared
>      system catalog, but we don't want to force initdb during 8.1 beta.
>      This change lets us cleanly load existing dump files even if they
>      contain outright wrong information about a PL's support functions, such
>      as a wrong path to the shared library or a missing validator function.
>      Also, we can revert the recent kluges to make pg_dump dump PL support
>      functions that are stored in pg_catalog.  While at it, I removed the
>      code in pg_regress that replaced $libdir with a hardcoded path for
>      temporary installations.  This is no longer needed given our support
>      for relocatable installations.
>
> This moved the helper functions into pg_catalog, but the author probably
> didn't realize that public schema helper functions would continue to be
> dumped by pg_dump.  These helper functions continued to be
> dumped/restored until the rename.  There are certainly helper functions
> for other languages that are still duplicated in the public schema ---
> there is nothing unique about plpython.  We are only seeing problems
> because of the plpython.so rename.

Yes, I had the same layout for plpgsql. Already got rid of the public entry.

>
> In normal use, a pg_dumpall restore would throw an error about a missing
> helper function shared objects, but the pg_catalog entry would continue
> to work just fine.  Odd we have not heard complaints from users seeing
> that error on restore --- odds are, they are just ignoring the error,
> which pg_upgrade does not do.
>
> The attached pg_upgrade patch adds checks for this plpython helper
> function and reports a proper error, suggesting how to fix the problem:
>
>     Performing Consistency Checks
>     -----------------------------
>     Checking current, bin, and data directories                 ok
>     Checking cluster versions                                   ok
>
>     The old cluster has a "plpython_call_handler" function defined
>     in the "public" schema which is a duplicate of the one defined
>     in the "pg_catalog" schema.  You can confirm this by executing
>     in psql:
>
>             \df *.plpython_call_handler
>
>     The "public" schema version of this function was created by a
>     pre-8.1 install of plpython, and must be removed for pg_upgrade
>     to complete because it references a now-obsolete "plpython"
>     shared object file.  You can remove the "public" schema version
>     of this function by running the following command:
>
>             DROP FUNCTION public.plpython_call_handler()
>
>     in each affected database:
>
>             test
>             test3
>
>     Remove the problem functions from the old cluster to continue.
>     Failure, exiting

Thanks.

>
> We could do the same for other PL languages if they are ever renamed.  I
> suppose we don't care about fixing the duplicate schema entries.

Not sure what duplicate entries you are referring to. The two pg_catalog
entries refer to different handlers, the call handler and the inline
handler.

>


--
Adrian Klaver
adrian.klaver@gmail.com

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Bruce Momjian
Date:
On Tue, May 29, 2012 at 07:13:24PM -0700, Adrian Klaver wrote:
> >This moved the helper functions into pg_catalog, but the author probably
> >didn't realize that public schema helper functions would continue to be
> >dumped by pg_dump.  These helper functions continued to be
> >dumped/restored until the rename.  There are certainly helper functions
> >for other languages that are still duplicated in the public schema ---
> >there is nothing unique about plpython.  We are only seeing problems
> >because of the plpython.so rename.
>
> Yes, I had the same layout for plpgsql. Already got rid of the public entry.

OK, confirmed then.

> >We could do the same for other PL languages if they are ever renamed.  I
> >suppose we don't care about fixing the duplicate schema entries.
>
> Not sure what duplicate entries you are referring to. The two
> pg_catalog entries refer to different handlers, the call handler and
> the inline handler.

My point is that plpython_call_handler is defined in the public and
pg_catalog schema, as are other language handlers.

In fact, an argument could be made that the bug is really in pg_dump.
When we moved the language handlers into pg_catalog, I don't think any
effort was made to suppress the public schema handlers from being dumped.
Maybe that's where the fix should be.  It would allow us to fix all the
languages, not just plpython when using pg_upgrade.  I think the big
question is can we uniquely identify them.

I did see this C comment in pg_dump.c:

    /*
     * Determine whether we want to dump definitions for procedural languages.
     * Since the languages themselves don't have schemas, we can't rely on
     * the normal schema-based selection mechanism.  We choose to dump them
     * whenever neither --schema nor --table was given.  (Before 8.1, we used
     * the dump flag of the PL's call handler function, but in 8.1 this will
     * probably always be false since call handlers are created in pg_catalog.)
     *
     * For some backwards compatibility with the older behavior, we forcibly
     * dump a PL if its handler function (and validator if any) are in a
     * dumpable namespace.  That case is not checked here.
     *
     * Also, if the PL belongs to an extension, we do not use this heuristic.
     * That case isn't checked here either.
     */
    static bool
    shouldDumpProcLangs(void)


--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Bruce Momjian
Date:
On Tue, May 29, 2012 at 10:34:16PM -0400, Bruce Momjian wrote:
> My point is that plpython_call_handler is defined in the public and
> pg_catalog schema, as are other language handlers.
>
> In fact, an argument could be made that the bug is really in pg_dump.
> When we moved the language handlers into pg_catalog, I don't think any
> effort was made to suppress the public schema handlers from being dumped.
> Maybe that's where the fix should be.  It would allow us to fix all the
> languages, not just plpython when using pg_upgrade.  I think the big
> question is can we uniquely identify them.
>
> I did see this C comment in pg_dump.c:
>
>     /*
>      * Determine whether we want to dump definitions for procedural languages.
>      * Since the languages themselves don't have schemas, we can't rely on
>      * the normal schema-based selection mechanism.  We choose to dump them
>      * whenever neither --schema nor --table was given.  (Before 8.1, we used
>      * the dump flag of the PL's call handler function, but in 8.1 this will
>      * probably always be false since call handlers are created in pg_catalog.)
>      *
>      * For some backwards compatibility with the older behavior, we forcibly
>      * dump a PL if its handler function (and validator if any) are in a
>      * dumpable namespace.  That case is not checked here.
>      *
>      * Also, if the PL belongs to an extension, we do not use this heuristic.
>      * That case isn't checked here either.
>      */
>     static bool
>     shouldDumpProcLangs(void)

OK, so what do people want me to do on this?  Apply my pg_upgrade fix or
go for a more general fix that will prevent pg_dump from dumping out
these duplicate functions --- it would involve checking for public
schema functions who's names and probin match pg_pltemplate entries.
Either will fix pg_upgrade.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Adrian Klaver
Date:
On 05/31/2012 11:53 AM, Bruce Momjian wrote:
> On Tue, May 29, 2012 at 10:34:16PM -0400, Bruce Momjian wrote:

>
> OK, so what do people want me to do on this?  Apply my pg_upgrade fix or
> go for a more general fix that will prevent pg_dump from dumping out
> these duplicate functions --- it would involve checking for public
> schema functions who's names and probin match pg_pltemplate entries.
> Either will fix pg_upgrade.
>

I would say the pg_dump fix. That one gets rid of the duplicates for
everyone, not just those folks using pg_upgrade.

--
Adrian Klaver
adrian.klaver@gmail.com

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Tom Lane
Date:
Adrian Klaver <adrian.klaver@gmail.com> writes:
> On 05/31/2012 11:53 AM, Bruce Momjian wrote:
>> On Tue, May 29, 2012 at 10:34:16PM -0400, Bruce Momjian wrote:
>> OK, so what do people want me to do on this?  Apply my pg_upgrade fix or
>> go for a more general fix that will prevent pg_dump from dumping out
>> these duplicate functions --- it would involve checking for public
>> schema functions who's names and probin match pg_pltemplate entries.
>> Either will fix pg_upgrade.

> I would say the pg_dump fix. That one gets rid of the duplicates for
> everyone, not just those folks using pg_upgrade.

Hm, I'm not sure about that.  The general charter of pg_dump is to
produce a dump that will replicate the state of the database.
Editorializing on it in order to make it more likely to reload in a
different version of PG seems to violate that charter.

I think the current state where pg_upgrade just complains about those
functions and tells you to remove them by hand is far safer than
creating blind spots in pg_dump.

            regards, tom lane

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Bruce Momjian
Date:
On Thu, May 31, 2012 at 06:24:04PM -0400, Tom Lane wrote:
> Adrian Klaver <adrian.klaver@gmail.com> writes:
> > On 05/31/2012 11:53 AM, Bruce Momjian wrote:
> >> On Tue, May 29, 2012 at 10:34:16PM -0400, Bruce Momjian wrote:
> >> OK, so what do people want me to do on this?  Apply my pg_upgrade fix or
> >> go for a more general fix that will prevent pg_dump from dumping out
> >> these duplicate functions --- it would involve checking for public
> >> schema functions who's names and probin match pg_pltemplate entries.
> >> Either will fix pg_upgrade.
>
> > I would say the pg_dump fix. That one gets rid of the duplicates for
> > everyone, not just those folks using pg_upgrade.
>
> Hm, I'm not sure about that.  The general charter of pg_dump is to
> produce a dump that will replicate the state of the database.
> Editorializing on it in order to make it more likely to reload in a
> different version of PG seems to violate that charter.
>
> I think the current state where pg_upgrade just complains about those
> functions and tells you to remove them by hand is far safer than
> creating blind spots in pg_dump.

Agreed.  I think the big question is whether the 8.1 move of the PL
language support functions to pg_catalog should have suppressed dumping
the pre-8.1 PL functions in the public schema.

Another question is whether having these functions in two schemas
presents any possible danger.  Users using pg_dumpall and restoring (not
using pg_upgrade) will have the plpython functions removed because they
will error out, so maybe we should just let the plpython renaming trim
those out.  However, this doesn't remove the other PL lanauge
duplicates.

I share Tom's caution on this, but I think we need to make sure we are
addressing any possible risk of an isolated pg_upgrade fix, now that we
understand the cause.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Adrian Klaver
Date:
On 05/31/2012 03:30 PM, Bruce Momjian wrote:
> On Thu, May 31, 2012 at 06:24:04PM -0400, Tom Lane wrote:

>>
>> Hm, I'm not sure about that.  The general charter of pg_dump is to
>> produce a dump that will replicate the state of the database.
>> Editorializing on it in order to make it more likely to reload in a
>> different version of PG seems to violate that charter.
>>
>> I think the current state where pg_upgrade just complains about those
>> functions and tells you to remove them by hand is far safer than
>> creating blind spots in pg_dump.
>
> Agreed.  I think the big question is whether the 8.1 move of the PL
> language support functions to pg_catalog should have suppressed dumping
> the pre-8.1 PL functions in the public schema.
>
> Another question is whether having these functions in two schemas
> presents any possible danger.  Users using pg_dumpall and restoring (not
> using pg_upgrade) will have the plpython functions removed because they
> will error out, so maybe we should just let the plpython renaming trim
> those out.  However, this doesn't remove the other PL lanauge
> duplicates.
>
> I share Tom's caution on this, but I think we need to make sure we are
> addressing any possible risk of an isolated pg_upgrade fix, now that we
> understand the cause.

There are two different things at work here. One the duplicate
functions, two the renaming of the plpython libraries. The first would
not have been an issue without the second. I still say for compatibility
the plpython libraries should follow a scheme that reflects how the
language is being handled in the database:

plpython.so -> plpython2.so
plpython2.so
plpython3.so (optional)

Not sure if that is possible, just my 2 cents.

In any case the overriding issue would seem to be covered in another
thread on --hackers 'pg_upgrade libraries check' In other words
determining who is responsible for tracking library changes?

>


--
Adrian Klaver
adrian.klaver@gmail.com

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Bruce Momjian
Date:
On Thu, May 31, 2012 at 06:30:30PM -0400, Bruce Momjian wrote:
> > Hm, I'm not sure about that.  The general charter of pg_dump is to
> > produce a dump that will replicate the state of the database.
> > Editorializing on it in order to make it more likely to reload in a
> > different version of PG seems to violate that charter.
> >
> > I think the current state where pg_upgrade just complains about those
> > functions and tells you to remove them by hand is far safer than
> > creating blind spots in pg_dump.
>
> Agreed.  I think the big question is whether the 8.1 move of the PL
> language support functions to pg_catalog should have suppressed dumping
> the pre-8.1 PL functions in the public schema.
>
> Another question is whether having these functions in two schemas
> presents any possible danger.  Users using pg_dumpall and restoring (not
> using pg_upgrade) will have the plpython functions removed because they
> will error out, so maybe we should just let the plpython renaming trim
> those out.  However, this doesn't remove the other PL lanauge
> duplicates.
>
> I share Tom's caution on this, but I think we need to make sure we are
> addressing any possible risk of an isolated pg_upgrade fix, now that we
> understand the cause.

FYI, this query will show any functions defined in the public schema
who's names match pg_pltemplate helper functions:

    SELECT proname
    FROM pg_proc JOIN pg_namespace ON (pronamespace = pg_namespace.oid)
    WHERE proname IN
        (
            SELECT tmplhandler FROM pg_pltemplate
            UNION
            SELECT tmplinline FROM pg_pltemplate
            UNION
            SELECT tmplvalidator FROM pg_pltemplate
        ) AND
        nspname = 'public';

This is normal in pre-8.1 but might indicate orphaned functions in PG
8.1+.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Bruce Momjian
Date:
On Thu, May 31, 2012 at 09:37:06PM -0400, Bruce Momjian wrote:
> > I share Tom's caution on this, but I think we need to make sure we are
> > addressing any possible risk of an isolated pg_upgrade fix, now that we
> > understand the cause.
>
> FYI, this query will show any functions defined in the public schema
> who's names match pg_pltemplate helper functions:
>
>     SELECT proname
>     FROM pg_proc JOIN pg_namespace ON (pronamespace = pg_namespace.oid)
>     WHERE proname IN
>         (
>             SELECT tmplhandler FROM pg_pltemplate
>             UNION
>             SELECT tmplinline FROM pg_pltemplate
>             UNION
>             SELECT tmplvalidator FROM pg_pltemplate
>         ) AND
>         nspname = 'public';
>
> This is normal in pre-8.1 but might indicate orphaned functions in PG
> 8.1+.

OK, based on the lack of excitement in doing anything more invasive, I
have applied the pg_upgrade fix to PG 9.2 for the plpython helper
function appearing in the public schema and referencing an old shared
library.

Should this be back-patched?  (Problem first reported in PG 9.1 due to
the removal of a backward-compatibility symlink.)  I am not sure we have
had enough problem reports to warrant it (3 reports).

(Should bug discussions stay on the bugs list?  I did for this bug.)

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Robert Haas
Date:
On Fri, Jun 1, 2012 at 11:48 AM, Bruce Momjian <bruce@momjian.us> wrote:
> Should this be back-patched? =A0(Problem first reported in PG 9.1 due to
> the removal of a backward-compatibility symlink.) =A0I am not sure we have
> had enough problem reports to warrant it (3 reports).

We back-patch a lot of things based on a single report.  Three is a lot.

--=20
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: BUG #6666: pg_upgrade 9.2beta1 plpython/plpython2

From
Bruce Momjian
Date:
On Fri, Jun 08, 2012 at 11:32:40AM -0400, Robert Haas wrote:
> On Fri, Jun 1, 2012 at 11:48 AM, Bruce Momjian <bruce@momjian.us> wrote:
> > Should this be back-patched?  (Problem first reported in PG 9.1 due to
> > the removal of a backward-compatibility symlink.)  I am not sure we have
> > had enough problem reports to warrant it (3 reports).
>
> We back-patch a lot of things based on a single report.  Three is a lot.

OK, batckpatched to Postgres 9.1.  I don't think anyone is still major
release upgrading to 9.0 or earlier.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +