Thread: Switching PL/Python to Python 3 by default in PostgreSQL 12

Switching PL/Python to Python 3 by default in PostgreSQL 12

From
Steven Pousty
Date:
Greetings:
I am not sure if this has been brought up before but Python 2 is EOL on Jan 1 2020. After that time there will not be any security fixes or patches.


According to our most recent official documentation:
https://www.postgresql.org/docs/11/plpython-python23.html

" The default will probably be changed to Python 3 in a distant future release of PostgreSQL, depending on the progress of the migration to Python 3 in the Python community."

I know we are late in the Postgresql 12 cycle but I think switching the default to Python 3 is warranted given:
1. The serious nature of not having a default supported Python version soon after the PostgreSQL 12 release
2. The next opportunity to change the default will be late 2020

If we do not switch our default version and a vulnerability arises in Python 2 then we will end up either
1. Telling our users to run the default PL/Python with a known security vulnerability
2. The PostgreSQL community patching it's python

I know there are implications for swapping the default version but I think that is outweighed by the seriousness of the situation.

Thanks
Steve

Re: Switching PL/Python to Python 3 by default in PostgreSQL 12

From
Peter Eisentraut
Date:
On 2019-07-06 21:02, Steven Pousty wrote:
> /" The default will probably be changed to Python 3 in a distant future
> release of PostgreSQL, depending on the progress of the migration to
> Python 3 in the Python community."/
>
> I know we are late in the Postgresql 12 cycle but I think switching the
> default to Python 3 is warranted given:

"The default" in the above statement refers to what version the language
name "plpythonu" invokes.  That is separate from whether an installer or
packager chooses to install a Python-2-based version.  I expect that as
operating systems remove Python 2 from their systems, packagers will
also remove Python-2-based plpython packages.  So there is no issue.

We could change plpythonu to be an alias for plpython3u instead of
plpython2u, but that would be a PG13 (or later) change.  I notice that
PEP-394 now supports this.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



Re: Switching PL/Python to Python 3 by default in PostgreSQL 12

From
Steven Pousty
Date:
Why would it be a 13 or later issue?
I am specifically talking about changing the default. 

On Sat, Jul 6, 2019, 6:28 PM Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote:
On 2019-07-06 21:02, Steven Pousty wrote:
> /" The default will probably be changed to Python 3 in a distant future
> release of PostgreSQL, depending on the progress of the migration to
> Python 3 in the Python community."/
>
> I know we are late in the Postgresql 12 cycle but I think switching the
> default to Python 3 is warranted given:

"The default" in the above statement refers to what version the language
name "plpythonu" invokes.  That is separate from whether an installer or
packager chooses to install a Python-2-based version.  I expect that as
operating systems remove Python 2 from their systems, packagers will
also remove Python-2-based plpython packages.  So there is no issue.

We could change plpythonu to be an alias for plpython3u instead of
plpython2u, but that would be a PG13 (or later) change.  I notice that
PEP-394 now supports this.

--
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Switching PL/Python to Python 3 by default in PostgreSQL 12

From
Peter Eisentraut
Date:
On 2019-07-07 00:34, Steven Pousty wrote:
> Why would it be a 13 or later issue?

Because PostgreSQL 12 is feature frozen and in beta, and this issue is
not a regression.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



Re: Switching PL/Python to Python 3 by default in PostgreSQL 12

From
Tom Lane
Date:
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 2019-07-07 00:34, Steven Pousty wrote:
>> Why would it be a 13 or later issue?

> Because PostgreSQL 12 is feature frozen and in beta, and this issue is
> not a regression.

More to the point: it does not seem to me that we should change what
"plpythonu" means until Python 2 is effectively extinct in the wild.
Which is surely some years away yet.  If we change it sooner than
that, the number of people complaining that we broke perfectly good
installations will vastly outweigh the number of people who are
happy because we saved them one keystroke per function definition.

As a possibly relevant comparison, I get the impression that most
packagers of Python are removing the versionless "python" executable
name and putting *nothing* in its place.  You have to write python2
or python3 nowadays.  Individuals might still be setting up symlinks
so that "python" does what they want, but it's not happening at the
packaging/distro level.

(This comparison suggests that maybe what we should be thinking
about is a way to make it easier to change what "plpythonu" means
at the local-opt-in level.)

            regards, tom lane



Re: Switching PL/Python to Python 3 by default in PostgreSQL 12

From
Steven Pousty
Date:
The point of the links I sent from the Python community is that they wanted Python extinct in the wild as of Jan 1 next year. They are never fixing it, even for a security vulnerability.

It seems to me we roll out breaking changes with major versions. So yes, if the user chooses to upgrade to 12 and they haven't migrated their code to Python 2 it might not work. 

I don't have a good answer to no changes except regressions. I do hope, given how much our users expect us to be secure, that we weigh the consequences of making our default Python a version which is dead to the community a month or so after Postgresql 12s release. We can certainly take the stance of leave the Python version be, but it seems that we should then come up with a plan if there is a security vulnerability found in Python 2 after Jan 1st 2020. 

If Python 2 wasn't our default choice then I would be much more comfortable letting this just pass without mention. 

All that aside, I think allowing the admin set the default version of plpythonu to be an excellent idea. 

Thanks 
Steve 



On Sun, Jul 7, 2019, 8:26 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 2019-07-07 00:34, Steven Pousty wrote:
>> Why would it be a 13 or later issue?

> Because PostgreSQL 12 is feature frozen and in beta, and this issue is
> not a regression.

More to the point: it does not seem to me that we should change what
"plpythonu" means until Python 2 is effectively extinct in the wild.
Which is surely some years away yet.  If we change it sooner than
that, the number of people complaining that we broke perfectly good
installations will vastly outweigh the number of people who are
happy because we saved them one keystroke per function definition.

As a possibly relevant comparison, I get the impression that most
packagers of Python are removing the versionless "python" executable
name and putting *nothing* in its place.  You have to write python2
or python3 nowadays.  Individuals might still be setting up symlinks
so that "python" does what they want, but it's not happening at the
packaging/distro level.

(This comparison suggests that maybe what we should be thinking
about is a way to make it easier to change what "plpythonu" means
at the local-opt-in level.)

                        regards, tom lane

Re: Switching PL/Python to Python 3 by default in PostgreSQL 12

From
Peter Eisentraut
Date:
On 2019-07-07 21:26, Steven Pousty wrote:
> The point of the links I sent from the Python community is that they
> wanted Python extinct in the wild as of Jan 1 next year. They are never
> fixing it, even for a security vulnerability.

The operating systems that most of our users are going to run PostgreSQL
12 on will keep maintaining Python 2 for the foreseeable future.

I'm not trying to dismiss the importance of managing the Python
transition.  But this issue has been known for many years, and the
current setup is more or less in line with the wider world.  For
example, the Debian release that came out over the weekend still ships
with /usr/bin/python being Python 2.  So it is neither timely nor urgent
to try to make some significant change about this in PostgreSQL 12 right
now.  I would welcome patches for this for PostgreSQL 13.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



Re: Switching PL/Python to Python 3 by default in PostgreSQL 12

From
Tom Lane
Date:
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> I'm not trying to dismiss the importance of managing the Python
> transition.  But this issue has been known for many years, and the
> current setup is more or less in line with the wider world.  For
> example, the Debian release that came out over the weekend still ships
> with /usr/bin/python being Python 2.  So it is neither timely nor urgent
> to try to make some significant change about this in PostgreSQL 12 right
> now.  I would welcome patches for this for PostgreSQL 13.

I don't think it's been mentioned in this thread yet, but we *did*
recently install a configure-time preference for python3 over python2:

Author: Peter Eisentraut <peter@eisentraut.org>
Branch: master Release: REL_12_BR [7291733ac] 2019-01-13 10:23:48 +0100
Branch: REL_11_STABLE Release: REL_11_2 [3d498c65a] 2019-01-13 10:24:21 +0100
Branch: REL_10_STABLE Release: REL_10_7 [cd1873160] 2019-01-13 10:25:23 +0100

    configure: Update python search order

    Some systems don't ship with "python" by default anymore, only
    "python3" or "python2" or some combination, so include those in the
    configure search.

    Discussion:
https://www.postgresql.org/message-id/flat/1457.1543184081%40sss.pgh.pa.us#c9cc1199338fd6a257589c6dcea6cf8d

configure's search order is now $PYTHON, python, python3, python2.
I think it will be a very long time, if ever, before there would be
a reason to consider changing that.  Both of the first two options
represent following a clear user preference.

So the only thing that's really at stake is when/whether we can make
"plpythonu" a synonym for "plpython3u" rather than "plpython2u".
As I said already, I think that's got to be a long way off, since the
whole problem here is that python3 isn't a drop-in replacement for
python2.  We're much more likely to break existing functions than do
anything useful by forcibly switching the synonym.

But I could support having a way for individual installations to change
what the synonym means locally.  Perhaps we could think about how to do
that in conjunction with the project of getting rid of pg_pltemplate
that's been kicked around before [1][2][3].

            regards, tom lane

[1] https://www.postgresql.org/message-id/flat/763f2fe4-743f-d530-8831-20811edd3d6a%402ndquadrant.com

[2] https://www.postgresql.org/message-id/flat/7495.1524861244%40sss.pgh.pa.us

[3] https://www.postgresql.org/message-id/flat/5351890.TdMePpdHBD%40nb.usersys.redhat.com



Re: Switching PL/Python to Python 3 by default in PostgreSQL 12

From
Tom Lane
Date:
I wrote:
> But I could support having a way for individual installations to change
> what the synonym means locally.  Perhaps we could think about how to do
> that in conjunction with the project of getting rid of pg_pltemplate
> that's been kicked around before [1][2][3].

... actually, if we had that (i.e., languages fully defined by extensions
with no help from pg_pltemplate), wouldn't this be nearly trivial?
I'm imagining two extensions, one that defines plpythonu to call the
python2 code and one that defines it to call the python3 code, and you
install whichever you want.  They're separate from the extensions that
define plpython2u and plpython3u, so mix and match as you wish.

            regards, tom lane