Thread: obsoleting plpython2u and defaulting plpythonu to plpython3u
Forwarding to hackers. Hope that's OK after few days.
Pavel
---------- Forwarded Message ----------
Subject: obsoleting plpython2u and defaulting plpythonu to plpython3u
Date: Thursday, April 19, 2018, 11:56:40 AM CEST
From: Pavel Raiskup <praiskup@redhat.com>
To: pgsql-general <pgsql-general@postgresql.org>
Per current plpython docs:
The language named plpythonu implements PL/Python based on the default
Python language variant, which is currently Python 2. (This default is
independent of what any local Python installations might consider to
be their “default”, for example, what /usr/bin/python might be.) The
default will probably be changed to Python 3 in a distant future
release of PostgreSQL, depending on the progress of the migration to
Python 3 in the Python community.
.. the status quo seems to be bit optimistic with the "distant future",
and we should start thinking about dropping plpython2 support, same as
upstream (a bit optimistically too, IMO) does [1].
The docs don't suggest the explicit use of plpython2, but still the docs
are not discouraging from it -- so it is likely some clusters run against
that.
What's the expected future migration path from plpython2 to plpython3 in
such cases? I'm thinking about rewrite of the docs and creating some
scripting which could simplify the migration steps. Would such patches be
welcome at this point?
[1] https://pythonclock.org/
Pavel
-----------------------------------------
On 4/24/18 07:13, Pavel Raiskup wrote: > .. the status quo seems to be bit optimistic with the "distant future", > and we should start thinking about dropping plpython2 support, same as > upstream (a bit optimistically too, IMO) does [1]. I don't think we are going to drop Python 2 support anytime soon. > What's the expected future migration path from plpython2 to plpython3 in > such cases? I'm thinking about rewrite of the docs and creating some > scripting which could simplify the migration steps. Would such patches be > welcome at this point? I'm not sure what you have in mind. In many cases, you can just change the LANGUAGE clause. I suppose you could run 2to3 over the function body? Write a PL/Python function to run 2to3? -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Thanks Peter, On Wednesday, April 25, 2018 12:17:05 AM CEST Peter Eisentraut wrote: > On 4/24/18 07:13, Pavel Raiskup wrote: > > .. the status quo seems to be bit optimistic with the "distant future", > > and we should start thinking about dropping plpython2 support, same as > > upstream (a bit optimistically too, IMO) does [1]. > > I don't think we are going to drop Python 2 support anytime soon. Well, no hurry, just saying that we should start thinking about it. Mostly to have proper plan in advance because production support of plpython2 will be at least risky once Python 2 is EOL... Linux distributions are spending a lot of efforts to make Python 3 the default; and the plan is to drop Python 2 support entirely if possible. So full drop is not what ask for now (it will be needed for lts/enterprise distros for quite some time anyway), I just think that plpython3 should be the default (by build system, by 'plpythonu'=>'plpython3u', etc.). And maybe we could plan "deprecating" the plpython2 first in future major releases (not really 11 or 12, I have nothing concrete in mind now). > > What's the expected future migration path from plpython2 to plpython3 in > > such cases? I'm thinking about rewrite of the docs and creating some > > scripting which could simplify the migration steps. Would such patches be > > welcome at this point? > > I'm not sure what you have in mind. In many cases, you can just change > the LANGUAGE clause. I suppose you could run 2to3 over the function > body? Write a PL/Python function to run 2to3? Something along those lines, yes. Well, I don't have a real plan. I'd like to discuss what's the expected way out, how to give a chance to users to migrate from plpython2 to plpython3 _now_ (since both are yet available). I _guess_ the users don't care much about 2 vs. 3 in plpython - so they usually simply live with implicit plpython version - which would OTOH mean that the majority will have to face the migration sooner or later. Pavel
On 4/25/18 04:50, Pavel Raiskup wrote: > So full drop is not what ask for now (it will be needed for lts/enterprise > distros for quite some time anyway), I just think that plpython3 should be > the default (by build system, by 'plpythonu'=>'plpython3u', etc.). I don't think we should do that unless there is an update to PEP 394. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 4/25/18 04:50, Pavel Raiskup wrote:
>> So full drop is not what ask for now (it will be needed for lts/enterprise
>> distros for quite some time anyway), I just think that plpython3 should be
>> the default (by build system, by 'plpythonu'=>'plpython3u', etc.).
> I don't think we should do that unless there is an update to PEP 394.
PEP 394 points out that some distros (at least Arch) have already switched
"python" to mean "python3", and I gather from Pavel's inquiry that the day
is in sight when Fedora will do that. Perhaps we should think about
providing a configure switch to control whether "plpythonu" means
"plpython2u" or "plpython3u", so that distros can reasonably easily sync
PG's behavior with whatever they've chosen to do with the python shell
command. I agree that it's not PG's business to be out in front of this
change, but maybe we shouldn't be holding it back, either.
I'm not very sure how many moving parts would be involved in making that
happen. One thing we should probably do first is retire the pg_pltemplate
system catalog in favor of making PL extension scripts self-contained,
as you'd mentioned recently would be a good project to finish off.
regards, tom lane
On 04/26/2018 04:39 PM, Tom Lane wrote: > Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes: >> On 4/25/18 04:50, Pavel Raiskup wrote: >>> So full drop is not what ask for now (it will be needed for lts/enterprise >>> distros for quite some time anyway), I just think that plpython3 should be >>> the default (by build system, by 'plpythonu'=>'plpython3u', etc.). >> I don't think we should do that unless there is an update to PEP 394. > PEP 394 points out that some distros (at least Arch) have already switched > "python" to mean "python3", and I gather from Pavel's inquiry that the day > is in sight when Fedora will do that. Perhaps we should think about > providing a configure switch to control whether "plpythonu" means > "plpython2u" or "plpython3u", so that distros can reasonably easily sync > PG's behavior with whatever they've chosen to do with the python shell > command. I agree that it's not PG's business to be out in front of this > change, but maybe we shouldn't be holding it back, either. > > I'm not very sure how many moving parts would be involved in making that > happen. One thing we should probably do first is retire the pg_pltemplate > system catalog in favor of making PL extension scripts self-contained, > as you'd mentioned recently would be a good project to finish off. > > After installing python3, I switched buildfarm animal prion to build and test with python3 instead of python2 a day or two ago. All this required was setting PYTHON=/usr/bin/python3 in the environment. Everything else Just Worked. cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> After installing python3, I switched buildfarm animal prion to build and
> test with python3 instead of python2 a day or two ago. All this required
> was setting PYTHON=/usr/bin/python3 in the environment. Everything else
> Just Worked.
That just means we've hacked the regression tests to the point where
they work in that environment ;-). An installation from that code
would still think plpythonu means plpython2u.
regards, tom lane
On Thu, Apr 26, 2018 at 4:39 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > I'm not very sure how many moving parts would be involved in making that > happen. One thing we should probably do first is retire the pg_pltemplate > system catalog in favor of making PL extension scripts self-contained, > as you'd mentioned recently would be a good project to finish off. Incidentally, it recently came to my notice that we're still issuing hints that say: Use CREATE LANGUAGE to load the language into the database. That is now, and has been for some time, the wrong advice. We should really be recommending CREATE EXTENSION. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 4/26/18 16:39, Tom Lane wrote: >> I don't think we should do that unless there is an update to PEP 394. > > PEP 394 points out that some distros (at least Arch) have already switched > "python" to mean "python3", My recollection is that PEP 394 was written because Arch jumped the gun and the PEP is saying "do this instead". > and I gather from Pavel's inquiry that the day > is in sight when Fedora will do that. According to the plan I found at <https://fedoraproject.org/wiki/FinalizingFedoraSwitchtoPython3>, they will also wait for an update to PEP 394 ("and we will drive that update if needed"). So that's all great. There are some opportunities for prep work in the meantime that have been mentioned. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Robert Haas <robertmhaas@gmail.com> writes:
> Incidentally, it recently came to my notice that we're still issuing
> hints that say:
> Use CREATE LANGUAGE to load the language into the database.
> That is now, and has been for some time, the wrong advice. We should
> really be recommending CREATE EXTENSION.
Right. It's too late in the v11 cycle to consider any of this other work
right now, but I think it'd be reasonable to adjust those message(s) for
v11 ... any objections? I see two places in functioncmds.c that ought
to change, as well as a few examples in the SGML docs.
regards, tom lane
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes: > On 4/26/18 16:39, Tom Lane wrote: >> and I gather from Pavel's inquiry that the day >> is in sight when Fedora will do that. > According to the plan I found at > <https://fedoraproject.org/wiki/FinalizingFedoraSwitchtoPython3>, they > will also wait for an update to PEP 394 ("and we will drive that update > if needed"). Note that there's a specific timeline there: they expect PEP 394 to get changed not more than 2 years from now. I am not sure if Red Hat has enough pull in the Python community to guarantee that will happen, but do you want to bet against it? > So that's all great. There are some opportunities for prep work in the > meantime that have been mentioned. We should evidently plan on making this stuff happen in v12, or we're going to be behind the curve. regards, tom lane
On Friday, April 27, 2018 5:51:57 PM CEST Tom Lane wrote: > Note that there's a specific timeline there: they expect PEP 394 to get > changed not more than 2 years from now. I am not sure if Red Hat has > enough pull in the Python community to guarantee that will happen, but > do you want to bet against it? Well, also it depends what's meant by "get PEP 394 changed". My guess is that distros might well avoid providing /bin/python if that PEP is a concern.. that's actually typical situation in Fedora even now; python2 package isn't installed by default. And it can disappear entirely if that's too expensive for maintenance. To me, PEP 394 is mostly about python distributors - not about python projects; so IMO for PostgreSQL that PEP is a bit orthogonal thing. For quite some time already, doing 'CREATE EXTENSION plpython' implies at least some (maybe serious) migration headaches in not so distant future. > > So that's all great. There are some opportunities for prep work in the > > meantime that have been mentioned. > > We should evidently plan on making this stuff happen in v12, or we're > going to be behind the curve. Agreed, but I'm still not sure what would be the expected 2->3 migration path. If that change happens in PG v12, people will migrate to that from v11 and there's the trouble. Would it technically possible to upgrade such clusters with pg_upgrade when 'plpythonu' silently goes from 'plpython2.so' to 'plpython3.so' behind the scenes? Or is it expected that people will migrate semi-automatically/manually from plpython2 to plpython3 on v11 - before hitting pg_upgrade? Pavel
Pavel Raiskup <praiskup@redhat.com> writes:
> On Friday, April 27, 2018 5:51:57 PM CEST Tom Lane wrote:
>> We should evidently plan on making this stuff happen in v12, or we're
>> going to be behind the curve.
> Agreed, but I'm still not sure what would be the expected 2->3 migration
> path. If that change happens in PG v12, people will migrate to that from
> v11 and there's the trouble. Would it technically possible to upgrade
> such clusters with pg_upgrade when 'plpythonu' silently goes from
> 'plpython2.so' to 'plpython3.so' behind the scenes?
> Or is it expected that people will migrate semi-automatically/manually
> from plpython2 to plpython3 on v11 - before hitting pg_upgrade?
Good questions. I think that the most useful thing we could do in the
near term is to cause functions declared with LANGUAGE 'plpythonu'
to be declared with LANGUAGE 'plpython2u' as-stored. It'd then be on
the user's head to migrate at some point, changing that to LANGUAGE
'plpython3u' along with any required changes to the function body.
(This parallels the remarks in the Fedora migration plan about
discouraging packages from referencing "python" with no version number.)
The time frame for doing that would depend on the user's tolerance for
running out-of-support language code and/or their platform's tolerance for
shipping same. I'd imagine it would be quite a long time before PG proper
would say we no longer support python2, but individual distros would stop
shipping those packages whenever their policy dictates.
If we go down that path, then 'plpythonu' isn't really a language anymore,
nor an extension --- it's just an alias that CREATE FUNCTION recognizes
and replaces. In the past we've had needs of that sort and generally just
hacked it with some hard-wired code (see, eg, the overdue-for-retirement
code in ResolveOpClass()). I'm tempted to propose that we handle this
case with something equally grotty, that is basically do this in
CreateFunction:
if (strcmp(language, "plpythonu") == 0)
language = DEFAULT_PYTHON_LANGUAGE;
where DEFAULT_PYTHON_LANGUAGE is defined as either "plpython2u" or
"plpython3u" according to a configure switch. You could imagine
building some significantly more complicated infrastructure, but
I doubt it would be worth the effort; how many other use-cases are
we likely to have?
If we agree that something like this is the way to go, I wonder if we
shouldn't squeeze it into v11. If we wait till v12, that's not going
to ship till late 2019, which is going to be getting uncomfortably
close to when PEP 394 might change if we believe Fedora's schedule.
regards, tom lane
On 2018-04-27 13:19:43 -0400, Tom Lane wrote: > If we agree that something like this is the way to go, I wonder if we > shouldn't squeeze it into v11. If we wait till v12, that's not going > to ship till late 2019, which is going to be getting uncomfortably > close to when PEP 394 might change if we believe Fedora's schedule. I think that'd be reasonable. Another alternative would be to have a 'plpython' extension that depends on plpython2. That'd require users to specify CASCADE when creating it, but that actually seems like it could be a useful hint... I think it's probably not worth going that route though, because reassigning objects from one extension to another is more work than reasonable... Greetings, Andres Freund
Andres Freund <andres@anarazel.de> writes:
> Another alternative would be to have a 'plpython' extension that depends
> on plpython2. That'd require users to specify CASCADE when creating it,
> but that actually seems like it could be a useful hint... I think it's
> probably not worth going that route though, because reassigning objects
> from one extension to another is more work than reasonable...
Yeah, there's a separate set of questions about what happens during
pg_upgrade of a database containing the existing plpythonu extension.
You could imagine hacking the dump/reload case by defining plpythonu
as an empty extension with a dependency on plpython2u (or, maybe
someday in future, a dependency on plpython3u). But that won't do
the trick for binary upgrades; we might need special-case code in
pg_dump/pg_upgrade to fix that.
regards, tom lane
On 2018-04-27 13:34:58 -0400, Tom Lane wrote: > Yeah, there's a separate set of questions about what happens during > pg_upgrade of a database containing the existing plpythonu extension. > > You could imagine hacking the dump/reload case by defining plpythonu > as an empty extension with a dependency on plpython2u (or, maybe > someday in future, a dependency on plpython3u). But that won't do > the trick for binary upgrades; we might need special-case code in > pg_dump/pg_upgrade to fix that. One way to deal with that is have an upgrade script that does the reassignment. Seems a mite cleaner than doing it in pg_dump. We could just have a query updating pg_depend to reference the new extension rather than the old. That'd mean we'd be in the "old" situation before somebody an extension update ran, but that seems hard to avoid? Greetings, Andres Freund
On 4/27/18 12:38, Pavel Raiskup wrote: > Well, also it depends what's meant by "get PEP 394 changed". My guess is > that distros might well avoid providing /bin/python if that PEP is > a concern.. that's actually typical situation in Fedora even now; python2 > package isn't installed by default. And it can disappear entirely if > that's too expensive for maintenance. I'm wondering who this situation is going to be resolved. Is anyone going to switch /usr/bin/python to Python 3 in the foreseeable future? A few have tried and regretted it. Or is /usr/bin/python just going to go away when Python 2 goes away? What we do in PostgreSQL would IMO depend on which of these choices is going to be the dominant one. If the former, then we can just switch plpythonu around to be Python 3 (perhaps a configure option). If the latter, then we don't need to do anything, except perhaps warn people to upgrade. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
And Gentoo too.
PEP 394 points out that some distros (at least Arch) have already switched
Pavel Raiskup <praiskup@redhat.com> writes:
> On Wednesday, April 25, 2018 12:17:05 AM CEST Peter Eisentraut wrote:
>> On 4/24/18 07:13, Pavel Raiskup wrote:
>>> What's the expected future migration path from plpython2 to plpython3 in
>>> such cases? I'm thinking about rewrite of the docs and creating some
>>> scripting which could simplify the migration steps. Would such patches be
>>> welcome at this point?
>> I'm not sure what you have in mind. In many cases, you can just change
>> the LANGUAGE clause. I suppose you could run 2to3 over the function
>> body? Write a PL/Python function to run 2to3?
> Something along those lines, yes. Well, I don't have a real plan. I'd
> like to discuss what's the expected way out, how to give a chance to users
> to migrate from plpython2 to plpython3 _now_ (since both are yet
> available).
> I _guess_ the users don't care much about 2 vs. 3 in plpython - so they
> usually simply live with implicit plpython version - which would OTOH mean
> that the majority will have to face the migration sooner or later.
It's still not very clear to me whether we can or should change anything
about what "plpythonu" means. However, it seems like we definitely ought
to be encouraging PL/Python users to switch their code to Python 3.
As an exercise, I wrote a little script that runs through all the Python 2
functions in a database and converts them to Python 3, insofar as "2to3"
can do that. This is by a noticeable margin the largest Python script
I've ever written, so I'm sure it sucks stylistically and perhaps
functionally too, but it will serve to get some discussion started.
Should we provide something like this to users, and if so how exactly
(installed file and/or documentation)? Would it be sane to squeeze
into v12 at this late date? I suspect that Fedora might want to have it
in their v12 distro even if we only put it into HEAD.
regards, tom lane
do $$
import re, subprocess, tempfile
# pattern to extract just the function header from pg_get_functiondef result
aspat = re.compile("^(.*?\nAS )", re.DOTALL)
# pattern for replacing LANGUAGE portion
langpat = re.compile("\n LANGUAGE plpython2?u\n")
# collect info about functions to update
rv = plpy.execute("""
select pg_catalog.pg_get_functiondef(p.oid) as fd, prosrc as body
from pg_catalog.pg_proc p join pg_catalog.pg_language l on p.prolang = l.oid
where lanname in ('plpythonu', 'plpython2u')
""")
# For some benighted reason, lib2to3 has exactly no documented API,
# so we must use the command-line API "2to3" instead. Adjust path
# and optional arguments for it here.
cmd2to3 = "2to3 --no-diffs"
# Make a temp directory to hold the file for it to work on.
with tempfile.TemporaryDirectory() as tmpdirname:
# process each function
for r in rv:
# extract everything but the body from pg_get_functiondef result
m = aspat.match(r["fd"])
if not m:
raise ValueError('unexpected match failure')
fheader = m.group(1)
# replace the language clause
fheader = langpat.sub("\n LANGUAGE plpython3u\n", fheader, 1)
# put body in a temp file so we can apply 2to3
f = open(tmpdirname + "/temp.py", mode = 'w')
f.write(r["body"])
f.close()
# apply 2to3 to body
subprocess.check_call(cmd2to3 + " -w " + tmpdirname + "/temp.py", shell=True)
f = open(tmpdirname + "/temp.py", mode = 'r')
fbody = f.read()
f.close()
# construct and execute SQL command to replace the function
newstmt = fheader + plpy.quote_literal(fbody)
# uncomment this for debugging purposes:
# plpy.info(newstmt)
plpy.execute(newstmt)
# commit after each successful replacement, in case a later one fails
plpy.commit()
$$
language plpython3u;