Thread: Time to drop plpython2?

Hi,

The last release of python2 was 2.7.18 released 2020-04-20, with support
already having ended before that 2020-01-01.

To me it seems time to drop plpython2 support. Supporting plpython2
until ~7 years after python2 is EOL is already quite long... It'd be one
thing if it were completely painless, but imo it's not.

I was about to list better plpython2/3 support (including the regex
replacements for the regress tests) as a TODO for the meson proposal,
but it doesn't really seem worth investing in that...

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> To me it seems time to drop plpython2 support. Supporting plpython2
> until ~7 years after python2 is EOL is already quite long... It'd be one
> thing if it were completely painless, but imo it's not.

7 years?  Oh, you're envisioning dropping plpython2 in HEAD but keeping
it alive in the back branches.  Yeah, it's a bit hard to justify
continuing support in HEAD.

            regards, tom lane

Hi,

On 2021-10-31 14:49:22 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > To me it seems time to drop plpython2 support. Supporting plpython2
> > until ~7 years after python2 is EOL is already quite long... It'd be one
> > thing if it were completely painless, but imo it's not.
> 
> 7 years?  Oh, you're envisioning dropping plpython2 in HEAD but keeping
> it alive in the back branches.

Yea. It's annoying to support python2, but I don't think it's severe
enough to consider dropping support in released branches. We might get
to that point for the newer released versions, but for now...


> Yeah, it's a bit hard to justify continuing support in HEAD.

Cool.

Greetings,

Andres Freund

On Mon, Nov 1, 2021 at 8:05 AM Andres Freund <andres@anarazel.de> wrote:
> > Yeah, it's a bit hard to justify continuing support in HEAD.

+1, it's dropping out of distros, it'd be unsupportable without
unreasonable effort.

On 31.10.21 19:45, Andres Freund wrote:
> To me it seems time to drop plpython2 support. Supporting plpython2
> until ~7 years after python2 is EOL is already quite long... It'd be one
> thing if it were completely painless, but imo it's not.
> 
> I was about to list better plpython2/3 support (including the regex
> replacements for the regress tests) as a TODO for the meson proposal,
> but it doesn't really seem worth investing in that...

Usually, we have dropped support for older Python versions when 
continued support would be a burden for some ongoing development 
project.  This would certainly count as one of those, I think.

I see you have posted a patch for this in the meson thread 
(https://www.postgresql.org/message-id/attachment/127770/v5-0003-plpython-Drop-support-python2.patch). 
  Here is my review of that.

I would change the search order in configure from

PGAC_PATH_PROGS(PYTHON, [python python3 python2])

to

PGAC_PATH_PROGS(PYTHON, [python3 python])

This makes sure you don't accidentally pick up a "python" binary that 
points to Python 2.  This would otherwise immediately generate lots of 
build failures on older platforms that still have Python 2 around.

You left two FIXME sections in plpython.h.  AFAICT, we can make the 
substitutions corresponding to those #define's in the source code and 
remove those blocks.

src/pl/plpython/expected/README should be updated for the removed files.

Some documentation updates are needed.  I see possibly relevant text in 
the following files:

hstore.sgml
install-windows.sgml
installation.sgml
json.sgml
libpq.sgml
plpython.sgml
ref/comment.sgml (examples)
ref/create_transform.sgml (examples)
ref/drop_transform.sgml (examples)
standalone-profile.xsl

src/tools/msvc/ has lots of Python-related content.

More stuff to clean up:

contrib/hstore_plpython/.gitignore
contrib/jsonb_plpython/.gitignore
contrib/ltree_plpython/.gitignore
src/pl/plpython/.gitignore

Finally, morally related, there is some Python 2/3 compat code in 
contrib/unaccent/generate_unaccent_rules.py that could be removed. 
Also, arguably, change the shebang line in that script.

Hi,

On 2021-11-04 19:58:54 +0100, Peter Eisentraut wrote:
> I see you have posted a patch for this in the meson thread
(https://www.postgresql.org/message-id/attachment/127770/v5-0003-plpython-Drop-support-python2.patch).

Yea, I was planning to post that here after a bit more polish. I mostly wanted
to get rid of the gross gross hack I did for the transmutation of the
regression test files.


> Here is my review of that.
> 
> I would change the search order in configure from
> 
> PGAC_PATH_PROGS(PYTHON, [python python3 python2])

> to
> 
> PGAC_PATH_PROGS(PYTHON, [python3 python])
> 
> This makes sure you don't accidentally pick up a "python" binary that points
> to Python 2.  This would otherwise immediately generate lots of build
> failures on older platforms that still have Python 2 around.

Yes, we should do that, at least for now. I did see build failures that
required me to specify the python version to avoid issues around it.


> You left two FIXME sections in plpython.h.  AFAICT, we can make the
> substitutions corresponding to those #define's in the source code and remove
> those blocks.

Yea, it shouldn't be hard. Just required more time than I had to send it out
before Nov 1st ;)

With meson I'd do a version: '>= 3' or such, to filter out a bare 'python'
being python2, but I don't think there's an equally trivial way to do that
with autoconf.


> Finally, morally related, there is some Python 2/3 compat code in
> contrib/unaccent/generate_unaccent_rules.py that could be removed. Also,
> arguably, change the shebang line in that script.

Hm. So far the python used for plpython and python for code generation etc is
independent. I don't know if plpython actually can be cross-compiled, but if
so, they'd have to be independent.  Otherwise I'd say we should just invoke
contrib/unaccent/generate_unaccent_rules.py with a python chosen by
configure/meson, rather than relying on a shebang that can't be adjusted
without modifying source code.


Another thing I wondered about is what we want to do with the extension
names. Do we want to leave it named plpython3u? Do we want to have a plpython
that depends on plpython3u?

I'd be inclined to just keep it at plpython3u for now, but there's an argument
that going for plpython would be better long term: Presumably there will be
python 4 at some point - but I'd expect that to not be a breaking release,
given the disaster that python 3 is. Making a non-versioned name better?

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> Another thing I wondered about is what we want to do with the extension
> names. Do we want to leave it named plpython3u? Do we want to have a plpython
> that depends on plpython3u?

I think we want to keep plpython3u.  Maybe we can point plpythonu
at that, but I'm concerned about the potential for user confusion.
In particular, I think there's a nonzero probability that someone
will choose to maintain plpython2u/plpythonu outside of core,
just because they still don't want to migrate their Python code.

> I'd be inclined to just keep it at plpython3u for now, but there's an argument
> that going for plpython would be better long term: Presumably there will be
> python 4 at some point - but I'd expect that to not be a breaking release,
> given the disaster that python 3 is. Making a non-versioned name better?

Meh.  If there is a python 4, I'd expect it to be named that way precisely
because it *is* a breaking release.  Why would we set ourselves up for
a repeat of this mess?

            regards, tom lane

On 04.11.21 20:54, Andres Freund wrote:
>> Finally, morally related, there is some Python 2/3 compat code in
>> contrib/unaccent/generate_unaccent_rules.py that could be removed. Also,
>> arguably, change the shebang line in that script.
> 
> Hm. So far the python used for plpython and python for code generation etc is
> independent. I don't know if plpython actually can be cross-compiled, but if
> so, they'd have to be independent.  Otherwise I'd say we should just invoke
> contrib/unaccent/generate_unaccent_rules.py with a python chosen by
> configure/meson, rather than relying on a shebang that can't be adjusted
> without modifying source code.

We don't rely on the shebang for running this (see Makefile).  I just 
see some trend of people changing shebang lines as a sort of signal, 
"this script has abandoned Python 2".  It's not very important.

> Another thing I wondered about is what we want to do with the extension
> names. Do we want to leave it named plpython3u? Do we want to have a plpython
> that depends on plpython3u?

I would tend to mirror what the Python community does with 
/usr/bin/python.  Right now, most people are removing /usr/bin/python 
and just provide /usr/bin/python3.  This would be analogous to what your 
patch does.  Some time in the future, they may add /usr/bin/python back, 
and then we could do the same, meaning add a plpythonu that depends on 
plpython3u.

... btw, there's a fairly critical gating factor for any plan to drop
python2 support: the buildfarm.  I just counted, and there are exactly
as many members running python 2.x as 3.x (49 apiece), not counting
Windows machines that aren't running configure.  We can't commit
something that's going to make half the buildfarm go red.

(It's likely that some fraction of them do already have python3 installed,
in which case the search order change Peter recommended would be enough to
fix it.  But I'm sure not all do.)

This ties into the business about converting the build system to meson,
as that also requires python 3 --- with, IIUC, a higher minimum version
than we might otherwise need.  I'm disinclined to cause two separate
flag days for buildfarm owners, so what I now think is we ought to put
this idea on the shelf until we've finished that conversion or decided
we're not gonna do it.  We need to identify exactly what needs to be
installed before we start pestering the owners.

            regards, tom lane

Andres Freund <andres@anarazel.de> writes:
> On 2021-11-14 21:24:31 -0500, Tom Lane wrote:
>> (It's likely that some fraction of them do already have python3 installed,
>> in which case the search order change Peter recommended would be enough to
>> fix it.  But I'm sure not all do.)

> How about committing the order change alone? That seems like something
> warranted completely in isolation? Afterwards we can see how many run what and
> go from there?

I don't think that's warranted.  The existing design is that we let
the user say which python is "python", and I do not think we should
change that in advance of actually dropping python2 support.

I was wondering about simply probing to see if python3 exists (and if
so, what version it is exactly), as an additional configure test that
doesn't hook into anything.  That would give us some information without
suddenly changing what is being tested.

            regards, tom lane

Andres Freund <andres@anarazel.de> writes:
> On 2021-11-15 12:19:51 -0500, Tom Lane wrote:
>> I was wondering about simply probing to see if python3 exists (and if
>> so, what version it is exactly), as an additional configure test that
>> doesn't hook into anything.  That would give us some information without
>> suddenly changing what is being tested.

> But this is probably a good compromise. Were you thinking of doing a proper
> autoconf test or just putting something like python3 --version || true in
> configure?

Hm, I had in mind an actual configure test; but since it's just a
temporary exploratory measure, shortcuts are fine.  However, I'm
not sure that what you suggest would result in capturing anything
in config.log.

> I guess it'd be easiest to interpret if we output the current PYTHON version
> and, iff < 3.0, also output the 'python3' version?

We'll already know if PYTHON is set, at least so far as the buildfarm
is concerned.

            regards, tom lane

On 15.11.21 19:18, Tom Lane wrote:
> Andres Freund<andres@anarazel.de>  writes:
>> On 2021-11-15 12:19:51 -0500, Tom Lane wrote:
>>> I was wondering about simply probing to see if python3 exists (and if
>>> so, what version it is exactly), as an additional configure test that
>>> doesn't hook into anything.  That would give us some information without
>>> suddenly changing what is being tested.
>> But this is probably a good compromise. Were you thinking of doing a proper
>> autoconf test or just putting something like python3 --version || true in
>> configure?
> Hm, I had in mind an actual configure test; but since it's just a
> temporary exploratory measure, shortcuts are fine.  However, I'm
> not sure that what you suggest would result in capturing anything
> in config.log.

I'm not sure this is really going to end up moving things forward.

I think we should just write to the build farm owners, we plan to drop 
python2 support in, say, 60 days, please update your setup to use 
python3 or disable python support.

If we add this test first, then all we're going to learn is probably 
that 60% of those who are currently using python2 don't have python3 
installed, and then we're still going to have to send that above email.

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
> If we add this test first, then all we're going to learn is probably 
> that 60% of those who are currently using python2 don't have python3 
> installed, and then we're still going to have to send that above email.

I don't know what fraction don't have python3 installed, and it doesn't
matter much, since it's unlikely that it's either 0% or 100%.  What
I hoped to learn was, of those who *do* have some python3 installed,
which version it is.  That might inform our thoughts about where to
set the minimum python3 version.

Relevant data points:

* Our docs claim the minimum 3.x version for pl/python itself is 3.1,
but this is unbacked by any testing; the oldest 3.x in the buildfarm
is 3.4.3 (three such animals).

* Meson only promises support back to python 3.6, but if that's
accurate it's going to be a problem for us, because there are lots
of live LTS distributions with older python3 (RHEL7, Solaris 11.3,
AIX 7.2 for starters).  I've been planning to do some testing and
see if meson will run under python 3.4 or 3.5.

But this probably belongs on the meson thread, since that looks like
it will be a much stronger constraint than pl/python is.

            regards, tom lane

Andres Freund <andres@anarazel.de> writes:
> On 2021-11-15 14:26:55 -0500, Tom Lane wrote:
>> * Meson only promises support back to python 3.6, but if that's
>> accurate it's going to be a problem for us, because there are lots
>> of live LTS distributions with older python3 (RHEL7, Solaris 11.3,
>> AIX 7.2 for starters).  I've been planning to do some testing and
>> see if meson will run under python 3.4 or 3.5.

> Slightly older versions, which do work to build postgres with the proposed
> patchset, run on python3 3.5. I don't think it's likely we could make the
> versions that only required 3.4 work reasonably.

OK, thanks for the datapoint.

> Is RHEL7 really an issue? I only have Centos 7 around, but that has python
> 3.6.

The info I checked said that RHEL7 originally shipped with 3.3.
I'm not sure that Red Hat would've outright replaced that, but they
do have a notion of add-on "software collections", and I'm certain
that they would have provided newer pythons via that mechanism.
So it should only be a big issue for someone who didn't want to install
an add-on collection.  I suppose the overlap of that group with the
people who will want to put PG 15+ on that platform is probably nil.

> I don't know much about AIX, but according to https://repology.org/project/python/versions
> the AIX toolbox has 3.7.11. I don't know enough about AIX to know whether
> there's other sources of python3, bison, etc that are common. On the AIX
> system I have access to they all seem to be symlinked to /opt/freeware, which
> I understand is that toolbox stuff?

Hmm, I was basing that on (a) what I can see installed on gcc119,
which is 3.5.1, and (b) AIX 7.2's 2015 release date, which matches up
pretty well with python 3.5.  As with RHEL, it's entirely likely that
IBM has made newer releases available as add-ons, but there's no
guarantee that any given installation would have that.

Solaris is a bit worse, since they shipped 11.3 a shade earlier,
with python 3.4.3.  No idea about add-on conventions on that
platform.

There's room to argue that all three of these will be out of
primary support before PG15 ships, so maybe we don't need to
worry about whether we can build with their default toolsets.
Still, it's a tradeoff I'd rather not make.

            regards, tom lane

On Mon, Nov 15, 2021 at 3:30 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> The info I checked said that RHEL7 originally shipped with 3.3.
> I'm not sure that Red Hat would've outright replaced that, but they
> do have a notion of add-on "software collections", and I'm certain
> that they would have provided newer pythons via that mechanism.
> So it should only be a big issue for someone who didn't want to install
> an add-on collection.  I suppose the overlap of that group with the
> people who will want to put PG 15+ on that platform is probably nil.

It'd only be an issue if they want to compile from source, right?
We're not speaking of changing the runtime prerequisites, IIUC.

I think it's really important that we continue to run on all of the
supported Linux distributions and even some recently-out-of-support
ones if they are popular. My experience is that many people stay on
whatever version of RHEL they've got for quite a long time. I think
we'll be doing ourselves a disservice if such people find that they
can't easily upgrade to newer versions of PostgreSQL without
installing a bunch of new software and/or upgrading software they've
already got on the machine.

On the other hand, the class of users that I'm thinking about does not
typically install anything from source, so they are not directly
impacted by build prerequisites.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Hi,

On November 15, 2021 12:36:11 PM PST, Robert Haas <robertmhaas@gmail.com> wrote:
>It'd only be an issue if they want to compile from source, right?
>We're not speaking of changing the runtime prerequisites, IIUC.

Correct.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Robert Haas <robertmhaas@gmail.com> writes:
> On Mon, Nov 15, 2021 at 3:30 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> The info I checked said that RHEL7 originally shipped with 3.3.

> It'd only be an issue if they want to compile from source, right?
> We're not speaking of changing the runtime prerequisites, IIUC.

I'm not sure.  Does it make sense to document that pl/python has
a different Python version requirement than the build system does?
If we do, who exactly is going to be testing that such a combination
works?  Will it even be possible to compile pl/python against Python
headers/libs of a different Python generation than meson is running
under?

ISTM we'd be a lot better off saying "the minimum Python version is
3.something", full stop, and then making sure that that minimum is
represented in the buildfarm.  But it's not quite clear yet what
"something" needs to be.

> I think it's really important that we continue to run on all of the
> supported Linux distributions and even some recently-out-of-support
> ones if they are popular.

I agree completely, which is why I'm raising the point.

            regards, tom lane

On 15.11.21 20:26, Tom Lane wrote:
> * Our docs claim the minimum 3.x version for pl/python itself is 3.1,
> but this is unbacked by any testing; the oldest 3.x in the buildfarm
> is 3.4.3 (three such animals).

I confirmed locally that 3.2.6 still works with PL/Python.  I expect 
that Python 3.1 also still works, except that I can't build it at the 
moment.  Python 3.0.* was sort of a dud, so it's not really interesting 
to consider.

Partially because of the need to keep supporting Python 2, there hasn't 
been any work done in PL/Python to make use of any newer APIs, so the 
Python 3 minimum version is frozen in time at the moment.

On Mon, Nov 15, 2021 at 4:12 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > It'd only be an issue if they want to compile from source, right?
> > We're not speaking of changing the runtime prerequisites, IIUC.
>
> I'm not sure.  Does it make sense to document that pl/python has
> a different Python version requirement than the build system does?
> If we do, who exactly is going to be testing that such a combination
> works?  Will it even be possible to compile pl/python against Python
> headers/libs of a different Python generation than meson is running
> under?

Hmm, that's true. I hadn't considered the fact that anyone who is
packaging PostgreSQL probably also wants to build plpython. However,
it's possible that a side-by-side install of a newer python version
could be used for the build system while building against the system
python for plpython. That might or might not be too exotic a
configuration for someone to consider.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

On 11/14/21 21:24, Tom Lane wrote:
> ... btw, there's a fairly critical gating factor for any plan to drop
> python2 support: the buildfarm.  I just counted, and there are exactly
> as many members running python 2.x as 3.x (49 apiece), not counting
> Windows machines that aren't running configure.  We can't commit
> something that's going to make half the buildfarm go red.
>
> (It's likely that some fraction of them do already have python3 installed,
> in which case the search order change Peter recommended would be enough to
> fix it.  But I'm sure not all do.)
>
> This ties into the business about converting the build system to meson,
> as that also requires python 3 --- with, IIUC, a higher minimum version
> than we might otherwise need.  I'm disinclined to cause two separate
> flag days for buildfarm owners, so what I now think is we ought to put
> this idea on the shelf until we've finished that conversion or decided
> we're not gonna do it.  We need to identify exactly what needs to be
> installed before we start pestering the owners.
>
>             


crake has been using 2.7, but has 3.9.7 installed. I tried switching to
that but ran into an issue with upgrading things from 9.5 on. It would
have been all the way back to 9.2 but the plpython tests drop the
extension even though the transform tests in contrib don't, and neither
do the plperl tests drop plperlu. I'm rather inclined to say we should
stop doing that, or at least be consistent about it.

Having rebuilt all the branches from 9.5 and up the cross version
upgrade tests are now passing on crake.

My other machine with an old python instance is bowerbird. It has python
3.4 installed but not used, alongside 2.7 which is udsed. I will install
the latest and see if that can be made to work.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On 11/16/21 11:26, Andrew Dunstan wrote:
>
> My other machine with an old python instance is bowerbird. It has python
> 3.4 installed but not used, alongside 2.7 which is udsed. I will install
> the latest and see if that can be made to work.
>
>

bowerbird is now building with python 3.10


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On 15.11.21 19:52, Peter Eisentraut wrote:
> I think we should just write to the build farm owners, we plan to drop 
> python2 support in, say, 60 days, please update your setup to use 
> python3 or disable python support.

This discussion stalled.  I think we should do *something* for 
PostgreSQL 15.

I suspect at this point, the Meson move isn't going to happen for 
PostgreSQL 15.  Even if the code were ready, which it is not, then this 
kind of thing would surely be something more sensible to put into PG16 
early rather than PG15 late.  So I'm setting aside any concerns about 
which Python 3.x is the appropriate minimum level.

There is the discussion [0], which involves raising the minimum Python 
version from 2.6 to 2.7, which would affect some build farm members. 
But if we're going to drop 2.x anyway for PG15, say, then we don't need 
to spend time organizing a 2.6 to 2.7 transition.  (Although 
backpatching might be required.)

Also, I was recently doing some bug fixing work in PL/Python, and just 
getting a Python 2 installation is not straightforward these days on the 
sort of OS that might be on a development machine.  So there is also 
that driver for dropping Python 2.

I think what I wrote in the message I'm responding to is the best way 
forward, although maybe with less than 60 days now.

Thoughts?


[0]: 
https://www.postgresql.org/message-id/flat/c74add3c-09c4-a9dd-1a03-a846e5b2fc52%40enterprisedb.com

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
> On 15.11.21 19:52, Peter Eisentraut wrote:
>> I think we should just write to the build farm owners, we plan to drop 
>> python2 support in, say, 60 days, please update your setup to use 
>> python3 or disable python support.

> This discussion stalled.  I think we should do *something* for 
> PostgreSQL 15.

Agreed.  We want plpython2 out of there for v15, or we'll be adding
another year to the length of time we're on the hook to support
python2 in the back branches --- which, as you say, is getting
ever more painful to do.

> I suspect at this point, the Meson move isn't going to happen for 
> PostgreSQL 15.

Also agreed.  Nonetheless, we need to make a recommendation to the
buildfarm owners about what's the minimum python3 version we intend
to support going forward.  Do we want to just set it at 3.6, with
the expectation that the meson move will happen before too long?

(I'm not going to accept being fuzzy on this point, because I need
to know what to install on prairiedog and gaur.  I intend to install
whatever our minimum supported version is, not only to keep us honest
on that being actually supported, but because I anticipate that
newer python versions will be correspondingly harder to build on
ancient platforms.)

            regards, tom lane

On 11.01.22 17:06, Tom Lane wrote:
> Nonetheless, we need to make a recommendation to the
> buildfarm owners about what's the minimum python3 version we intend
> to support going forward.  Do we want to just set it at 3.6, with
> the expectation that the meson move will happen before too long?

Well, the minimum supported version has always been the oldest version 
that actually works.  I don't think we ever said, we support >= X, even 
though < X still actually works, about any dependency.

I don't care much to tie this to Meson right now.  Meson might well move 
to 3.8 next week and ruin this whole scheme.

I'm okay with issuing some sort of recommendation for what is reasonable 
to test, and 3.5 or 3.6 seems like a good cutoff, considering what LTS 
OS currently ship.  But I'm not sure if that is the same as "support".

On Wed, Jan 12, 2022 at 2:39 AM Peter Eisentraut
<peter.eisentraut@enterprisedb.com> wrote:
> Well, the minimum supported version has always been the oldest version
> that actually works.  I don't think we ever said, we support >= X, even
> though < X still actually works, about any dependency.

I think that we sometimes say that versions < X are unsupported if we
are unable to test whether or not they work. In other words, I think
the relevant question is whether we are able to demonstrate that it
works, not whether it actually does work.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
> On 11.01.22 17:06, Tom Lane wrote:
>> Nonetheless, we need to make a recommendation to the
>> buildfarm owners about what's the minimum python3 version we intend
>> to support going forward.

> Well, the minimum supported version has always been the oldest version 
> that actually works.  I don't think we ever said, we support >= X, even 
> though < X still actually works, about any dependency.

The concern I have is how do we know what "actually works", if we're
not testing it?  installation.sgml currently promises python2 >= 2.6,
and we know that that works because we have 2.6 in the buildfarm.
It also promises python3 >= 3.1, but we have no buildfarm animals
testing anything older than 3.4.3, so I don't think that promise
is worth the electrons it's written on.  Furthermore, if the meson
conversion forces people to update their python3 to something newer,
there will probably be no testing of plpython against anything older
than what meson requires.

> I don't care much to tie this to Meson right now.  Meson might well move 
> to 3.8 next week and ruin this whole scheme.

Wouldn't be a problem unless our build scripts require that newer
version of meson.  Andres mentioned earlier that we should be able
to run with some older meson versions that only require python 3.5
or so, so I'm hoping we can end up with directives like "use meson
X or later and python 3.5 or later".

> I'm okay with issuing some sort of recommendation for what is reasonable 
> to test, and 3.5 or 3.6 seems like a good cutoff, considering what LTS 
> OS currently ship.  But I'm not sure if that is the same as "support".

Well, I'll see about putting 3.5 on my dinosaurs, and hope I don't
have to do it over.

Anyway, getting back to the point: I think we should notify the
owners ASAP and set a 30-day deadline.  We should try to get this
done before the March CF starts, so it's too late for a 60-day
grace period.  In any case, the worst-case scenario for an owner
is to disable --with-python until they have time to do an upgrade,
so it doesn't seem like a month is a big problem.

            regards, tom lane

On 12.01.22 19:49, Tom Lane wrote:
> Anyway, getting back to the point: I think we should notify the
> owners ASAP and set a 30-day deadline.  We should try to get this
> done before the March CF starts, so it's too late for a 60-day
> grace period.  In any case, the worst-case scenario for an owner
> is to disable --with-python until they have time to do an upgrade,
> so it doesn't seem like a month is a big problem.

Sure, let's do that.  I don't have a buildfarm animal these days, so I'm 
not on that list, so it would be great if you could do that

Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
> On 12.01.22 19:49, Tom Lane wrote:
>> Anyway, getting back to the point: I think we should notify the
>> owners ASAP and set a 30-day deadline.

> Sure, let's do that.  I don't have a buildfarm animal these days, so I'm 
> not on that list, so it would be great if you could do that

Done.  I told them "mid February", so we can plan on say the 15th
as the target date for pushing a patch.

I realized BTW that the meson business is not relevant for prairiedog
or gaur.  Those animals will die regardless of python version because
they can't build ninja (for lack of <spawn.h>).  So I think maybe
I'll install python 3.1 and see if that compatibility claim is really
true ;-)

            regards, tom lane

Hi,


On 2022-01-11 13:59:32 +0100, Peter Eisentraut wrote:
> I suspect at this point, the Meson move isn't going to happen for PostgreSQL
> 15.

Yes - I IIRC even noted early in the thread that I don't think it's realistic
to finish it in time for 15. There's just a good amount of portability hacking
left to be done, and a good chunk of cleanup. I have a new colleague working
on automating testing it on more platforms.

Getting a few of the prerequisite patches in would make it more likely to
succeed in 16 though.


> Even if the code were ready, which it is not, then this kind of thing
> would surely be something more sensible to put into PG16 early rather than
> PG15 late.

Right. There's also a lot of discussion around transition paths etc that need
to be happening first. A lot of other projects that moved ran with multiple
buildsystems for ~1 release, so things could mopped up without blocking
everyone.


> I think what I wrote in the message I'm responding to is the best way
> forward, although maybe with less than 60 days now.

Already happened, but I'm a belated +1 ;)

Greetings,

Andres Freund

I wrote:
> Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
>> On 12.01.22 19:49, Tom Lane wrote:
> Anyway, getting back to the point: I think we should notify the
> owners ASAP and set a 30-day deadline.

>> Sure, let's do that.  I don't have a buildfarm animal these days, so I'm 
>> not on that list, so it would be great if you could do that

> Done.  I told them "mid February", so we can plan on say the 15th
> as the target date for pushing a patch.

Well, it's mid-February.  Do we have a python2-removal patch
that's ready to go?

            regards, tom lane

Andres Freund <andres@anarazel.de> writes:
> On 2022-02-14 14:18:58 -0500, Tom Lane wrote:
>> Well, it's mid-February.  Do we have a python2-removal patch
>> that's ready to go?

> I can refresh mine. Iit might be good to first reapply
> f201da39edc - "Make configure prefer python3 to plain python."
> for a few days?

We could I guess, but does it really buy anything?  I'm sure that
some of the buildfarm still hasn't updated their Python installation,
but it'll be about the same failure we'd get from the final patch.

            regards, tom lane

Hi,

On February 14, 2022 12:48:12 PM PST, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>Andres Freund <andres@anarazel.de> writes:
>> On 2022-02-14 14:18:58 -0500, Tom Lane wrote:
>>> Well, it's mid-February.  Do we have a python2-removal patch
>>> that's ready to go?
>
>> I can refresh mine. Iit might be good to first reapply
>> f201da39edc - "Make configure prefer python3 to plain python."
>> for a few days?
>
>We could I guess, but does it really buy anything?  I'm sure that
>some of the buildfarm still hasn't updated their Python installation,
>but it'll be about the same failure we'd get from the final patch.

I guess what I was actually wondering about - but didn't write - was whether it's worth rejecting python 2 with just
configure/ msvc perl changes initially. 

The proper removal of python 2 support includes a bunch of buildsystem and code changes. Seems like it could be useful
tohave a snapshot of the buildfarm state after rejecting python 2, separate from the more verbose changes. 

Andres


--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Andres Freund <andres@anarazel.de> writes:
> On February 14, 2022 12:48:12 PM PST, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> We could I guess, but does it really buy anything?  I'm sure that
>> some of the buildfarm still hasn't updated their Python installation,
>> but it'll be about the same failure we'd get from the final patch.

> I guess what I was actually wondering about - but didn't write - was whether it's worth rejecting python 2 with just
configure/ msvc perl changes initially. 

Ah, I see.  +1, that would let us start nagging the laggard buildfarm
owners right away.  But I do think we want the rest of the cleanup
done pretty soon --- especially simplification of the plpython
regression test arrangements.

            regards, tom lane

On 15.02.22 22:40, Andres Freund wrote:
> 1: plpython: Reject Python 2 during build configuration.

There is a bit of redundancy in the new wording in installation.sgml:

"The minimum required version is <productname>Python</productname> 3.2 
or later."

There is also a small bit in install-windows.sgml that would be worth 
updating:

<programlisting>
$config->{python} = 'c:\python26';
</programlisting>

Other than that, this looks fine to me, and we should move forward with 
this now.

> 2: plpython: Remove plpythonu, plpython2u extensions.
> 3: plpython: Remove regression test infrastructure for Python 2.

The removal of the Python 2 contrib extension variants ended up in patch 
3, which seems incorrect.

Hi,

On 2022-02-16 15:43:19 +0100, Peter Eisentraut wrote:
> On 15.02.22 22:40, Andres Freund wrote:
> > 1: plpython: Reject Python 2 during build configuration.
> 
> There is a bit of redundancy in the new wording in installation.sgml:
> 
> "The minimum required version is <productname>Python</productname> 3.2 or
> later."

I stared for a bit, and I just don't see the redundancy?


> There is also a small bit in install-windows.sgml that would be worth
> updating:
> 
> <programlisting>
> $config->{python} = 'c:\python26';
> </programlisting>

Yea. I saw that but ended up with not updating it, because it already was out
of date ;). But I guess I'll just put 310 there or something.


> Other than that, this looks fine to me, and we should move forward with this
> now.

Cool, will apply 1) later today.


> > 2: plpython: Remove plpythonu, plpython2u extensions.
> > 3: plpython: Remove regression test infrastructure for Python 2.
> 
> The removal of the Python 2 contrib extension variants ended up in patch 3,
> which seems incorrect.

Uh, huh. That was a mistake, not sure what happened...

Greetings,

Andres Freund

On 02/16/22 14:58, Andres Freund wrote:
>> "The minimum required version is <productname>Python</productname> 3.2 or
>> later."
> 
> I stared for a bit, and I just don't see the redundancy?

"minimum ... or later" maybe?

Regards,
-Chap

On 2022-02-16 15:05:36 -0500, Chapman Flack wrote:
> On 02/16/22 14:58, Andres Freund wrote:
> >> "The minimum required version is <productname>Python</productname> 3.2 or
> >> later."
> > 
> > I stared for a bit, and I just don't see the redundancy?
> 
> "minimum ... or later" maybe?

Ah. Thanks.

On 2022-02-16 11:58:50 -0800, Andres Freund wrote:
> Cool, will apply 1) later today.

Done. Curious how red the BF will turn out to be. Let's hope it's not
too bad.

Hi,

On 2022-02-16 22:52:08 -0800, Andres Freund wrote:
> On 2022-02-16 11:58:50 -0800, Andres Freund wrote:
> > Cool, will apply 1) later today.
> 
> Done. Curious how red the BF will turn out to be. Let's hope it's not
> too bad.

I've pinged the owners of the animals failing so far:
- snakefly, massasauga
- jay, trilobite, hippopotamus, avocet
- myna, butterflyfish
- rhinoceros

Greetings,

Andres Freund

On Wed, Feb 16, 2022 at 11:14:46PM -0800, Andres Freund wrote:
> I've pinged the owners of the animals failing so far:
> - myna, butterflyfish

These two are managed by a colleague, and I have an access to them.
They should get back to green quickly now, if I did not mess up..
--
Michael

Hi,

On 2022-02-16 23:14:46 -0800, Andres Freund wrote:
> >
> > Done. Curious how red the BF will turn out to be. Let's hope it's not
> > too bad.
>
> I've pinged the owners of the animals failing so far:

Now also pinged:
- curculio
- guaibasaurus
- snapper
- gadwall, takin


> - snakefly, massasauga

Fixed.


> - jay, trilobite, hippopotamus, avocet

Nothing yet.


> - myna, butterflyfish

Fixed, as noted by Micheal on this thread.


> - rhinoceros

Joe replied that he is afk, looking into it tomorrow.


Greetings,

Andres Freund

On 2022-02-17 19:08, Andres Freund wrote:
>> I've pinged the owners of the animals failing so far:
> 
> Now also pinged:
> - curculio

Should be fixed by now.

I did install the python3-package but the binary was called:

/usr/local/bin/python3.5

for some reason so configure didn't pick it up.

Fixed it by creating a symlink to:

/usr/local/bin/python3

/Mikael

On 2/17/22 13:08, Andres Freund wrote:
> On 2022-02-16 23:14:46 -0800, Andres Freund wrote:
>> > Done. Curious how red the BF will turn out to be. Let's hope it's not
>> > too bad.

>> - rhinoceros
> 
> Joe replied that he is afk, looking into it tomorrow.

I installed python3 packages (initially forgetting the devel package -- 
d'oh!) and changed build-farm.conf thusly:

8<-------------------
***************
*** 185,190 ****
--- 185,193 ----

         build_env => {

+               # specify python 3
+               PYTHON => 'python3',
+
                 # use a dedicated cache for the build farm. this should 
give us
                 # very high hit rates and slightly faster cache searching.
                 #
8<-------------------

That seems to have worked.

But the question is, is that the correct/recommended method?

Joe
-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

Hi,

On 2022-02-18 14:19:49 -0500, Joe Conway wrote:
> On 2/17/22 13:08, Andres Freund wrote:
> > On 2022-02-16 23:14:46 -0800, Andres Freund wrote:
> > > > Done. Curious how red the BF will turn out to be. Let's hope it's not
> > > > too bad.
> 
> > > - rhinoceros
> > 
> > Joe replied that he is afk, looking into it tomorrow.
> 
> I installed python3 packages (initially forgetting the devel package --
> d'oh!) and changed build-farm.conf thusly:
> 
> 8<-------------------
> ***************
> *** 185,190 ****
> --- 185,193 ----
> 
>         build_env => {
> 
> +               # specify python 3
> +               PYTHON => 'python3',
> +
>                 # use a dedicated cache for the build farm. this should give
> us
>                 # very high hit rates and slightly faster cache searching.
>                 #
> 8<-------------------
> 
> That seems to have worked.
> 
> But the question is, is that the correct/recommended method?

If python3 is in PATH, then you shouldn't need that part.

Greetings,

Andres Freund

On 2/18/22 14:37, Andres Freund wrote:
> Hi,
> 
> On 2022-02-18 14:19:49 -0500, Joe Conway wrote:
>> On 2/17/22 13:08, Andres Freund wrote:
>> > On 2022-02-16 23:14:46 -0800, Andres Freund wrote:
>> > > > Done. Curious how red the BF will turn out to be. Let's hope it's not
>> > > > too bad.
>> 
>> > > - rhinoceros
>> > 
>> > Joe replied that he is afk, looking into it tomorrow.
>> 
>> I installed python3 packages (initially forgetting the devel package --
>> d'oh!) and changed build-farm.conf thusly:
>> 
>> 8<-------------------
>> ***************
>> *** 185,190 ****
>> --- 185,193 ----
>> 
>>         build_env => {
>> 
>> +               # specify python 3
>> +               PYTHON => 'python3',
>> +
>>                 # use a dedicated cache for the build farm. this should give
>> us
>>                 # very high hit rates and slightly faster cache searching.
>>                 #
>> 8<-------------------
>> 
>> That seems to have worked.
>> 
>> But the question is, is that the correct/recommended method?
> 
> If python3 is in PATH, then you shouldn't need that part.

Not quite -- python3 is definitely in the PATH:

8<-------------------
$ which python3
/usr/bin/python3
8<-------------------

And I gather that merely installing python3 RPMs on a RHEL-esque 7.X 
system does not replace the python symlink:

8<-------------------
$ yum whatprovides /usr/bin/python

python-2.7.5-89.el7.x86_64 : An interpreted, interactive, 
object-oriented programming language
Repo        : base
Matched from:
Filename    : /usr/bin/python

$ ll /usr/bin/python
lrwxrwxrwx. 1 root root 7 Mar 13  2021 /usr/bin/python -> python2
8<-------------------


Joe
-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

Hi,

On 2022-02-18 14:46:39 -0500, Joe Conway wrote:
> On 2/18/22 14:37, Andres Freund wrote:
> > > That seems to have worked.
> > > 
> > > But the question is, is that the correct/recommended method?
> > 
> > If python3 is in PATH, then you shouldn't need that part.
> 
> Not quite -- python3 is definitely in the PATH:
> 
> 8<-------------------
> $ which python3
> /usr/bin/python3
> 8<-------------------
> 
> And I gather that merely installing python3 RPMs on a RHEL-esque 7.X system
> does not replace the python symlink:
> 
> 8<-------------------
> $ yum whatprovides /usr/bin/python
> 
> python-2.7.5-89.el7.x86_64 : An interpreted, interactive, object-oriented
> programming language
> Repo        : base
> Matched from:
> Filename    : /usr/bin/python
> 
> $ ll /usr/bin/python
> lrwxrwxrwx. 1 root root 7 Mar 13  2021 /usr/bin/python -> python2
> 8<-------------------

Yea, that all looks fine. What's the problem if you don't specify the
PYTHON=python3? We try python3, python in that order by default, so it should
pick up the same python3 you specify explicitly?

Or maybe I'm just misunderstanding what you're asking...

Greetings,

Andres Freund

On 2/18/22 15:25, Andres Freund wrote:
> On 2022-02-18 14:46:39 -0500, Joe Conway wrote:
>> $ ll /usr/bin/python
>> lrwxrwxrwx. 1 root root 7 Mar 13  2021 /usr/bin/python -> python2
>> 8<-------------------
> 
> Yea, that all looks fine. What's the problem if you don't specify the
> PYTHON=python3? We try python3, python in that order by default, so it should
> pick up the same python3 you specify explicitly?

Initially I just installed the python3 RPMs and when I tried running 
manually it was still error'ing on configure due to finding python2.

Even after adding EXPORT PYTHON=python3 to my ~/.bash_profile I was 
seeing the same.

By adding PYTHON => 'python3' to build-farm.conf I saw that the error 
changed to indicate missing python3-devel package. Once I installed 
that, everything went green.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

Hi,

On 2022-02-18 15:35:37 -0500, Joe Conway wrote:
> Initially I just installed the python3 RPMs and when I tried running
> manually it was still error'ing on configure due to finding python2.

> Even after adding EXPORT PYTHON=python3 to my ~/.bash_profile I was seeing
> the same.
>
> By adding PYTHON => 'python3' to build-farm.conf I saw that the error
> changed to indicate missing python3-devel package. Once I installed that,
> everything went green.

Hm. It definitely did test python3, earlier today:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=rhinoceros&dt=2022-02-18%2016%3A52%3A13

checking for python3... no
checking for python... /usr/bin/python


the next run then saw:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=rhinoceros&dt=2022-02-18%2017%3A50%3A09
checking for PYTHON... python3
configure: using python 3.6.8 (default, Nov 16 2020, 16:55:22)
checking for Python sysconfig module... yes
checking Python configuration directory... /usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu
checking Python include directory... -I/usr/include/python3.6m

but then failed because the python headers weren't available:
checking for Python.h... no
configure: error: header file <Python.h> is required for Python


Note that this did *not* yet use PYTHON => 'python3' in build_env, but has it
in the environment starting the buildfarm client.


the next run succeeded, with 'PYTHON' => 'python3' in build env. But
presumably this just was because you installed the python3-devel package?

Greetings,

Andres Freund

On 2022-Feb-17, Andres Freund wrote:

> Now also pinged:
> - guaibasaurus

Fixed now (apt install python3-dev), but I had initially added
PYTHON=>python3 to the .conf, unsuccessfully because I failed to install
the dev pkg.  After the first success I removed that line.  It should
still work if we do test python3 first, but if it does fail, then I'll
put that line back.

Thanks

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/
"The problem with the future is that it keeps turning into the present"
(Hobbes)

On 2/18/22 15:53, Andres Freund wrote:
> the next run succeeded, with 'PYTHON' => 'python3' in build env. But
> presumably this just was because you installed the python3-devel package?


Ok, I guess I got confused when it failed due to the missing devel 
package, because I removed the PYTHON => 'python3' from the build env 
and it is still getting successfully past the configure stage. Sorry for 
the noise.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development

Hi,

Thanks to some more buildfarm animal updates things are looking better. I
think there's now only three owners that haven't updated their animals
successfully. One of which I hadn't yet pinged (chipmunk / Heikki), done now.

There's snapper ("pgbf [ a t ] twiska.com"), and there's Mark Wong's large
menagerie. Mark said yesterday that he's working on updating.


There's one further failure, but the symptoms are quite different. I've also
pinged its owner. I think it's a problem on the system, rather than our side,
but less certain than with the other cases:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=haddock&dt=2022-02-17%2023%3A36%3A09

checking Python.h usability... no
checking Python.h presence... yes
configure: WARNING: Python.h: present but cannot be compiled
configure: WARNING: Python.h:     check for missing prerequisite headers?
configure: WARNING: Python.h: see the Autoconf documentation
configure: WARNING: Python.h:     section "Present But Cannot Be Compiled"
configure: WARNING: Python.h: proceeding with the compiler's result
configure: WARNING:     ## ---------------------------------------------- ##
configure: WARNING:     ## Report this to pgsql-bugs@lists.postgresql.org ##
configure: WARNING:     ## ---------------------------------------------- ##
checking for Python.h... no
configure: error: header file <Python.h> is required for Python

configure:19158: ccache gcc -c -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla
-Wendif-labels-Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security
-fno-strict-aliasing-fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -g -m32
-I/usr/include/python3.9-DWAIT_USE_POLL -D_POSIX_PTHREAD_SEMANTICS -I/usr/include/libxml2  conftest.c >&5
 
In file included from /usr/include/python3.9/Python.h:8,
                 from conftest.c:235:
/usr/include/python3.9/pyconfig.h:1443: warning: "SIZEOF_LONG" redefined
 1443 | #define SIZEOF_LONG 8
      |
conftest.c:183: note: this is the location of the previous definition
  183 | #define SIZEOF_LONG 4
      |
In file included from /usr/include/python3.9/Python.h:8,
                 from conftest.c:235:
/usr/include/python3.9/pyconfig.h:1467: warning: "SIZEOF_SIZE_T" redefined
 1467 | #define SIZEOF_SIZE_T 8
      |
conftest.c:182: note: this is the location of the previous definition
  182 | #define SIZEOF_SIZE_T 4
      |
In file included from /usr/include/python3.9/Python.h:8,
                 from conftest.c:235:
/usr/include/python3.9/pyconfig.h:1476: warning: "SIZEOF_VOID_P" redefined
 1476 | #define SIZEOF_VOID_P 8
      |
conftest.c:181: note: this is the location of the previous definition
  181 | #define SIZEOF_VOID_P 4
      |
In file included from /usr/include/python3.9/Python.h:63,
                 from conftest.c:235:
/usr/include/python3.9/pyport.h:736:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc
config?)."
  736 | #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
      |  ^~~~~


This is a 64bit host, targetting 32bit "CFLAGS' => '-m32'. However it linked
successfully against python 2.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> There's one further failure, but the symptoms are quite different. I've also
> pinged its owner. I think it's a problem on the system, rather than our side,
> but less certain than with the other cases:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=haddock&dt=2022-02-17%2023%3A36%3A09

This one was discussed on the buildfarm-owners list last month.
There's no 32-bit python3 on that box, and apparently no plans
to install one --- it sounded like the box is due for retirement
anyway.

            regards, tom lane

Andres Freund <andres@anarazel.de> writes:
> On 2022-02-18 18:09:19 -0500, Tom Lane wrote:
>> This one was discussed on the buildfarm-owners list last month.
>> There's no 32-bit python3 on that box, and apparently no plans
>> to install one --- it sounded like the box is due for retirement
>> anyway.

> Any chance that was a private response? I just looked in the buildfarm-members
> list (I assume you meant that?), and didn't see anything:
> https://www.postgresql.org/message-id/flat/3162195.1642093011%40sss.pgh.pa.us

Oh ... hm, yeah, looking at my mail logs, that came in as private mail and
there's no corresponding buildfarm-members traffic.  I did not keep the
message unfortunately, but the owner indicated that he wasn't planning to
bother updating python.  Which is fine, but maybe we should press him to
remove --with-python instead of disabling the box altogether --- we don't
have a lot of Solaris clones in the farm.

            regards, tom lane

Hi everyone,

On Fri, Feb 18, 2022 at 02:41:04PM -0800, Andres Freund wrote:
> There's snapper ("pgbf [ a t ] twiska.com"), and there's Mark Wong's large
> menagerie. Mark said yesterday that he's working on updating.

I've made one pass.  Hopefully I didn't make any mistakes. :)

Regards,
Mark

On Thu, Feb 17, 2022 at 10:08:55AM -0800, Andres Freund wrote:
> On 2022-02-16 23:14:46 -0800, Andres Freund wrote:
>> I've pinged the owners of the animals failing so far:
>> - myna, butterflyfish
>
> Fixed, as noted by Micheal on this thread.

Fixed is an incorrect word here, "temporarily bypassed" fits better :)

Unfortunately, I had to remove --with-python on both animals for the
time being, as I was not able to figure out why Python.h could not be
found in those installations, and it was Friday afternoon.  I'll try
to investigate and re-enable that some time next week.
--
Michael

Hi,

On 2022-02-19 02:00:28 +0000, Mark Wong wrote:
> On Fri, Feb 18, 2022 at 02:41:04PM -0800, Andres Freund wrote:
> > There's snapper ("pgbf [ a t ] twiska.com"), and there's Mark Wong's large
> > menagerie. Mark said yesterday that he's working on updating.
> 
> I've made one pass.  Hopefully I didn't make any mistakes. :)

Unfortunately it looks like it wasn't quite enough. All, or nearly all, your
animals that ran since still seem to be failing in the same spot...

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gadwall&dt=2022-02-19%2011%3A22%3A48

checking Python.h usability... no
checking Python.h presence... no
checking for Python.h... no
configure: error: header file <Python.h> is required for Python


For that machine (and the other debian based ones) the relevant package likely
is python3-dev.

For the Red Hat and Suse ones, it's likely python3-devel.


I've wondered before if it's worth maintaining a list of packages for
dependencies for at least the more popular distros. It's annoying to have to
figure it out everytime one needs to test something.


FWIW, here's the recipe I just used to verify the packages necessary for
Python.h to be found:

$ podman run --rm -it opensuse/leap
# zypper install -y python3
# ls -l $(python3 -c "import sysconfig; print(sysconfig.get_config_var('INCLUDEPY'))")/Python.h
<file not found>
# zypper install -y python3-devel
# ls -l $(python3 -c "import sysconfig; print(sysconfig.get_config_var('INCLUDEPY'))")/Python.h
-rw-r--r-- 1 root root 3221 Jan  4 14:04 /usr/include/python3.6m/Python.h

(Wow, zypper repos are expensive to refresh. And I thought dnf was slow doing
so, compared to apt.)

Greetings,

Andres Freund

On Sat, Feb 19, 2022 at 08:22:29AM -0800, Andres Freund wrote:
> Hi,
> 
> On 2022-02-19 02:00:28 +0000, Mark Wong wrote:
> > On Fri, Feb 18, 2022 at 02:41:04PM -0800, Andres Freund wrote:
> > > There's snapper ("pgbf [ a t ] twiska.com"), and there's Mark Wong's large
> > > menagerie. Mark said yesterday that he's working on updating.
> > 
> > I've made one pass.  Hopefully I didn't make any mistakes. :)
> 
> Unfortunately it looks like it wasn't quite enough. All, or nearly all, your
> animals that ran since still seem to be failing in the same spot...
> 
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gadwall&dt=2022-02-19%2011%3A22%3A48
> 
> checking Python.h usability... no
> checking Python.h presence... no
> checking for Python.h... no
> configure: error: header file <Python.h> is required for Python
> 
> 
> For that machine (and the other debian based ones) the relevant package likely
> is python3-dev.
> 
> For the Red Hat and Suse ones, it's likely python3-devel.
> 
> 
> I've wondered before if it's worth maintaining a list of packages for
> dependencies for at least the more popular distros. It's annoying to have to
> figure it out everytime one needs to test something.
> 
> 
> FWIW, here's the recipe I just used to verify the packages necessary for
> Python.h to be found:
> 
> $ podman run --rm -it opensuse/leap
> # zypper install -y python3
> # ls -l $(python3 -c "import sysconfig; print(sysconfig.get_config_var('INCLUDEPY'))")/Python.h
> <file not found>
> # zypper install -y python3-devel
> # ls -l $(python3 -c "import sysconfig; print(sysconfig.get_config_var('INCLUDEPY'))")/Python.h
> -rw-r--r-- 1 root root 3221 Jan  4 14:04 /usr/include/python3.6m/Python.h
> 
> (Wow, zypper repos are expensive to refresh. And I thought dnf was slow doing
> so, compared to apt.)

Oops, made another pass for python3 dev libraries.

I can't seem to find archived ppc repos OpenSUSE Leap 43.2.  I'm
debating whether to disable python or upgrade/rebrand that animal for a
newer SUSE release.  I've stopped my cron jobs on this animal for the
time being.

Regards,
Mark

Hi,

On 2022-02-21 09:49:32 -0800, Mark Wong wrote:
> Oops, made another pass for python3 dev libraries.

Thanks!


> I can't seem to find archived ppc repos OpenSUSE Leap 43.2.  I'm
> debating whether to disable python or upgrade/rebrand that animal for a
> newer SUSE release.  I've stopped my cron jobs on this animal for the
> time being.

I assume you mean leap 42.3, that seemed to be the newest 4x.x? If so that's
been archived 2019-07-01 [1].

Leap's versioning is, uh, confusing. 13 -> 42 -> 15. Yea.

I don't think it's really useful to run out-of-support distribution
versions. Leaving security etc aside, not being able to install packages seems
sufficient reason to upgrade.

Greetings,

Andres Freund

[1]
https://lists.opensuse.org/archives/list/security-announce@lists.opensuse.org/message/6CIQPV3H6J4AIIXUKUGI4IESMTHIFFFB/

Mark Wong <markwkm@postgresql.org> writes:
> On Sat, Feb 19, 2022 at 08:22:29AM -0800, Andres Freund wrote:
>> Unfortunately it looks like it wasn't quite enough. All, or nearly all, your
>> animals that ran since still seem to be failing in the same spot...

> Oops, made another pass for python3 dev libraries.

You might need to do one more thing, which is manually blow away the cache
files under $BUILDFARM/accache.  For example, on demoiselle everything
looks fine in HEAD, but the back branches are failing like this:

checking for python... (cached) /usr/bin/python
./configure: line 10334: /usr/bin/python: No such file or directory
configure: using python
./configure: line 10342: test: : integer expression expected
checking for Python sysconfig module... no
configure: error: sysconfig module not found

Very recent versions of the buildfarm script will discard accache
automatically after a configure or make failure, but I think the
REL_11 you're running here doesn't have that defense.  It'll only
flush accache after a change in the configure script in git.

            regards, tom lane

Mark Wong <markwkm@postgresql.org> writes:
> Take 3. :)

> I've upgraded everyone to the v14 buildfarm scripts and made sure the
> --test passed on HEAD on each one.  So I hopefully didn't miss any
> (other than the one EOL OpenSUSE version that I will plan on upgrading.)

Thanks!

However ... it seems like most of your animals haven't run at all
in the last couple of days.  Did you maybe forget to re-enable
their cron jobs after messing with them, or something like that?

            regards, tom lane

On Tue, Feb 22, 2022 at 08:50:07PM -0500, Tom Lane wrote:
> Mark Wong <markwkm@postgresql.org> writes:
> > Take 3. :)
> 
> > I've upgraded everyone to the v14 buildfarm scripts and made sure the
> > --test passed on HEAD on each one.  So I hopefully didn't miss any
> > (other than the one EOL OpenSUSE version that I will plan on upgrading.)
> 
> Thanks!
> 
> However ... it seems like most of your animals haven't run at all
> in the last couple of days.  Did you maybe forget to re-enable
> their cron jobs after messing with them, or something like that?

Uh oh, more user error.  I tested run_build but run_branches wasn't
happy.  I'll start working through that...

Regards,
Mark

On Wed, Feb 23, 2022 at 07:59:01AM -0800, Mark Wong wrote:
> On Tue, Feb 22, 2022 at 08:50:07PM -0500, Tom Lane wrote:
> > Mark Wong <markwkm@postgresql.org> writes:
> > > Take 3. :)
> > 
> > > I've upgraded everyone to the v14 buildfarm scripts and made sure the
> > > --test passed on HEAD on each one.  So I hopefully didn't miss any
> > > (other than the one EOL OpenSUSE version that I will plan on upgrading.)
> > 
> > Thanks!
> > 
> > However ... it seems like most of your animals haven't run at all
> > in the last couple of days.  Did you maybe forget to re-enable
> > their cron jobs after messing with them, or something like that?
> 
> Uh oh, more user error.  I tested run_build but run_branches wasn't
> happy.  I'll start working through that...

I think I have most of them operational again.  I see some animals are
still failing on some branches, but still better overall?

I discovered that clang for gadwall and pintail got purged when I
removed python2, so I disabled those two animals.

Regards,
Mark

Hi,

On 2022-02-16 22:52:08 -0800, Andres Freund wrote:
> On 2022-02-16 11:58:50 -0800, Andres Freund wrote:
> > Cool, will apply 1) later today.
>
> Done. Curious how red the BF will turn out to be. Let's hope it's not
> too bad.

Now that the BF has stabilized, I've rebased and cleaned up the patches I'd
posted earlier. Attached for the first time is my attempt at cleaning up the
docs.

0003, the removal of code level support for Python 2, is now a good bit bigger
bigger, due to the removal of the last remnants of the Py2/3 porting layer.

I did so far leave in the "major version conflict" detection stuff in
plpy_main.c - that could again be useful? I'm leaning towards removing it, I'd
hope that there's not again such a painful transition, and we have the git
history if needed.

Regards,

Andres

Hi,

On 2022-03-06 17:30:15 -0800, Andres Freund wrote:
> 0003, the removal of code level support for Python 2, is now a good bit bigger
> bigger, due to the removal of the last remnants of the Py2/3 porting layer.

Oh, huh. Something here seems to be broken, causing a crash on windows, but
not elsewhere.

https://cirrus-ci.com/task/5288088032247808
stack trace and regression.diffs:

https://api.cirrus-ci.com/v1/artifact/task/5288088032247808/crashlog/crashlog-postgres.exe_1768_2022-03-07_01-07-48-535.txt
https://api.cirrus-ci.com/v1/artifact/task/5288088032247808/log/src/pl/plpython/regression.diffs

The previous version ran successfully on windows too. So I probably screwed
something up when removing the remaining Py2/3 porting layer bits... The fact
that it fails on windows but not elsewhere perhaps suggests it's something
around the width of long?

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> Now that the BF has stabilized, I've rebased and cleaned up the patches I'd
> posted earlier. Attached for the first time is my attempt at cleaning up the
> docs.

I looked through this quickly, and have a couple of nitpicks.  The
PGFILEDESC value for jsonb_plpython is worded randomly differently
from those for hstore_plpython and ltree_plpython.  I think we should
make it match those.  I also noted a couple of typos in the docs patch.
See attached delta patch (I named it .txt in hopes of not confusing
the cfbot).

I kind of wonder if we still need "46.1. Python 2 vs. Python 3" at
all.  It certainly doesn't seem like it still deserves its position
of honor as the first subsection.  Perhaps move it down to be the
last subsection?

Also, grepping reveals that vcregress.pl still has two stray references to
"plpythonu".  I did not touch that here, but maybe that has something
to do with the ci failure?

> I did so far leave in the "major version conflict" detection stuff in
> plpy_main.c - that could again be useful? I'm leaning towards removing it, I'd
> hope that there's not again such a painful transition, and we have the git
> history if needed.

I think we should leave it in.  I foresee that somebody will want to build
plpython2u as an out-of-core extension, at least for a few years yet.
If they do, and the core language does not have its half of that guard,
it'd be bad.

            regards, tom lane

diff --git a/contrib/jsonb_plpython/Makefile b/contrib/jsonb_plpython/Makefile
index e3f160510d..fea7bdfc00 100644
--- a/contrib/jsonb_plpython/Makefile
+++ b/contrib/jsonb_plpython/Makefile
@@ -4,7 +4,7 @@ MODULE_big = jsonb_plpython$(python_majorversion)
 OBJS = \
     $(WIN32RES) \
     jsonb_plpython.o
-PGFILEDESC = "jsonb_plpython - transform between jsonb and plpython3u"
+PGFILEDESC = "jsonb_plpython - jsonb transform for plpython"

 PG_CPPFLAGS = -I$(top_srcdir)/src/pl/plpython $(python_includespec)
-DPLPYTHON_LIBNAME='"plpython$(python_majorversion)"'

diff --git a/doc/src/sgml/json.sgml b/doc/src/sgml/json.sgml
index b3b16b8a77..c4223fafb6 100644
--- a/doc/src/sgml/json.sgml
+++ b/doc/src/sgml/json.sgml
@@ -716,7 +716,7 @@ UPDATE table_name SET jsonb_field[1]['a'] = '1';
   </para>

   <para>
-   The extensions for PL/Python is called <literal>jsonb_plpython3u</literal>.
+   The extension for PL/Python is called <literal>jsonb_plpython3u</literal>.
    If you use it, <type>jsonb</type> values are mapped to Python
    dictionaries, lists, and scalars, as appropriate.
   </para>
diff --git a/doc/src/sgml/ltree.sgml b/doc/src/sgml/ltree.sgml
index 4ff384c7ee..508f404ae8 100644
--- a/doc/src/sgml/ltree.sgml
+++ b/doc/src/sgml/ltree.sgml
@@ -826,15 +826,15 @@ ltreetest=> SELECT ins_label(path,2,'Space') FROM test WHERE path <@ 'Top.
   <title>Transforms</title>

   <para>
-   The <literal>ltree_plpython3u</literal> implement transforms for the
-   <type>ltree</type> type for PL/Python. If installed and specified when
+   The <literal>ltree_plpython3u</literal> extension implements transforms for
+   the <type>ltree</type> type for PL/Python. If installed and specified when
    creating a function, <type>ltree</type> values are mapped to Python lists.
    (The reverse is currently not supported, however.)
   </para>

   <caution>
    <para>
-    It is strongly recommended that the transform extensions be installed in
+    It is strongly recommended that the transform extension be installed in
     the same schema as <filename>ltree</filename>.  Otherwise there are
     installation-time security hazards if a transform extension's schema
     contains objects defined by a hostile user.
diff --git a/doc/src/sgml/plpython.sgml b/doc/src/sgml/plpython.sgml
index fab4330cd1..7698d5f068 100644
--- a/doc/src/sgml/plpython.sgml
+++ b/doc/src/sgml/plpython.sgml
@@ -52,7 +52,7 @@
   <para>
    PL/Python supports only Python 3. Past versions of
    <productname>PostgreSQL</productname> supported Python 2, using the
-   <literal>plpyhonu</literal> and <literal>plpython2u</literal> language
+   <literal>plpythonu</literal> and <literal>plpython2u</literal> language
    names.
   </para>
  </sect1>

Hi,

On 2022-03-07 15:05:03 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > Now that the BF has stabilized, I've rebased and cleaned up the patches I'd
> > posted earlier. Attached for the first time is my attempt at cleaning up the
> > docs.
> 
> I looked through this quickly, and have a couple of nitpicks.  The
> PGFILEDESC value for jsonb_plpython is worded randomly differently
> from those for hstore_plpython and ltree_plpython.  I think we should
> make it match those.

Makes sense.


> I also noted a couple of typos in the docs patch.
> See attached delta patch

Thanks, incorporated.


> I kind of wonder if we still need "46.1. Python 2 vs. Python 3" at
> all.  It certainly doesn't seem like it still deserves its position
> of honor as the first subsection.  Perhaps move it down to be the
> last subsection?

Agreed. I think it's useful to have it, just so we have a place mentioning
plpython[2]u. I put it as the second last subsection for now, somehow seemed
to belong before the env variable list? But...


> Also, grepping reveals that vcregress.pl still has two stray references to
> "plpythonu".

I left that in somewhat intentionally, it seemed the cleanest way to remove
plpython2 from vcregress.pl. It's purely cosmetic afaics.

A related question is whether we want to remove $(python_majorversion)
references in the makefiles?


> I did not touch that here, but maybe that has something to do with the ci
> failure?

The CI failure was caused by me screwing up search and replace
:(. Accidentally replaced PyString_AsString with PyUnicode_AsString instead of
P*L*yUnicode_AsString.

Somewhat surprised at that causing problems only on windows (even valgrind
didn't, although the required suppressions might have been the cause of
that). But well, better there than nowhere...


> > I did so far leave in the "major version conflict" detection stuff in
> > plpy_main.c - that could again be useful? I'm leaning towards removing it, I'd
> > hope that there's not again such a painful transition, and we have the git
> > history if needed.
> 
> I think we should leave it in.  I foresee that somebody will want to build
> plpython2u as an out-of-core extension, at least for a few years yet.
> If they do, and the core language does not have its half of that guard,
> it'd be bad.

Makes sense. Added a comment to that effect.


I noticed a few references to PLyString in function names. Probably should
replace them with PLyUnicode. Didn't do that yet in the attached...


Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> A related question is whether we want to remove $(python_majorversion)
> references in the makefiles?

I wouldn't.  I'm doubtful of your theory that there will never be
a Python 4.

This version seems ready-to-go to me, or at least ready to see
what the buildfarm makes of it.

            regards, tom lane

On 2022-03-07 20:11:52 -0500, Tom Lane wrote:
> This version seems ready-to-go to me, or at least ready to see
> what the buildfarm makes of it.

Pushed. Let's see...

Andres Freund <andres@anarazel.de> writes:
> On 2022-03-07 20:11:52 -0500, Tom Lane wrote:
>> This version seems ready-to-go to me, or at least ready to see
>> what the buildfarm makes of it.

> Pushed. Let's see...

wrasse says you were too quick to drop plpython_error_5.out.
Otherwise looks pretty good so far.

            regards, tom lane

Hi,

On 2022-03-07 23:39:39 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2022-03-07 20:11:52 -0500, Tom Lane wrote:
> >> This version seems ready-to-go to me, or at least ready to see
> >> what the buildfarm makes of it.
> 
> > Pushed. Let's see...
> 
> wrasse says you were too quick to drop plpython_error_5.out.

Does look like it. I'll try to find a distribution with an old python...


> Otherwise looks pretty good so far.

crake also failed. Looks like plpy_plpymodule.h needs to include plpython.h. A
pre-existing issue that just didn't happen to cause problems...

Greetings,

Andres Freund

Hi,

On 2022-03-07 20:59:16 -0800, Andres Freund wrote:
> On 2022-03-07 23:39:39 -0500, Tom Lane wrote:
> > Andres Freund <andres@anarazel.de> writes:
> > wrasse says you were too quick to drop plpython_error_5.out.
> 
> Does look like it. I'll try to find a distribution with an old python...

debian 8 did the trick.

The output from older python was in plpython_error.out the output from newer
python in plpython_error_5.out. Looked like python2 specific output.

Pushed.

I guess I should have grokked the _5 suffix - but the _3 suffix was used for
python3 specific content, so it wasn't obvious. Should we rename the file to
plpython_error_35.out?

A bit depressing to have a 500 line alternative output file for a one line
diff :(.


> > Otherwise looks pretty good so far.
> 
> crake also failed. Looks like plpy_plpymodule.h needs to include plpython.h. A
> pre-existing issue that just didn't happen to cause problems...

Fixed that.

Greetings,

Andres Freund

Andres Freund <andres@anarazel.de> writes:
> A bit depressing to have a 500 line alternative output file for a one line
> diff :(.

Yeah.  How badly do we need that particular test case?

            regards, tom lane

Hi,

On 2022-03-08 13:49:15 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > A bit depressing to have a 500 line alternative output file for a one line
> > diff :(.
>
> Yeah.  How badly do we need that particular test case?

A bit hard to tell. The test was introduced in

commit 2bd78eb8d51cc9ee03ba0287b23ff4c266dcd9b9
Author: Peter Eisentraut <peter_e@gmx.net>
Date:   2011-04-06 22:36:06 +0300

    Add traceback information to PL/Python errors

    This mimics the traceback information the Python interpreter prints
    with exceptions.

    Jan Urbański

and contains this ominous line:

+/* AttributeError at toplevel used to give segfaults with the traceback
+*/

Greetings,

Andres Freund

Hi,

On 2022-03-08 10:42:31 -0800, Andres Freund wrote:
> > crake also failed. Looks like plpy_plpymodule.h needs to include plpython.h. A
> > pre-existing issue that just didn't happen to cause problems...
> 
> Fixed that.

Hm. Now crake failed in XversionUpgrade-REL9_2_STABLE-HEAD:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2022-03-08%2018%3A47%3A22

except that the log doesn't actually indicate any problem? Andrew, any hint?

Greetings,

Andres Freund

On 08.03.22 20:03, Andres Freund wrote:
> Hi,
> 
> On 2022-03-08 13:49:15 -0500, Tom Lane wrote:
>> Andres Freund <andres@anarazel.de> writes:
>>> A bit depressing to have a 500 line alternative output file for a one line
>>> diff :(.
>>
>> Yeah.  How badly do we need that particular test case?
> 
> A bit hard to tell. The test was introduced in
> 
> commit 2bd78eb8d51cc9ee03ba0287b23ff4c266dcd9b9
> Author: Peter Eisentraut <peter_e@gmx.net>
> Date:   2011-04-06 22:36:06 +0300
> 
>      Add traceback information to PL/Python errors

We would probably try to write this test differently today, but at this 
point I wouldn't bother and just wait for Python 3.5 to fall off the end 
of the conveyor belt.

On 3/8/22 15:02, Andres Freund wrote:
> Hi,
>
> On 2022-03-08 10:42:31 -0800, Andres Freund wrote:
>>> crake also failed. Looks like plpy_plpymodule.h needs to include plpython.h. A
>>> pre-existing issue that just didn't happen to cause problems...
>> Fixed that.
> Hm. Now crake failed in XversionUpgrade-REL9_2_STABLE-HEAD:
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2022-03-08%2018%3A47%3A22
>
> except that the log doesn't actually indicate any problem? Andrew, any hint?
>


That was a snafu on my part, as I was adding extension upgrade / amcheck
testing. It's fixed now, so please disregard this one.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com