Thread: RHEL 8.0 build
Hi, Trying to set up a buildfarm animal on a RHEL 8.0 (beta) system, the build fails early: $ ./run_build.pl --nosend --nostatus --verbose Sat Nov 24 20:09:28 2018: buildfarm run for CHANGEME:HEAD starting CHANGEME:HEAD [20:09:28] checking out source ... CHANGEME:HEAD [20:09:33] checking if build run needed ... CHANGEME:HEAD [20:09:34] copying source to pgsql.build ... CHANGEME:HEAD [20:09:35] running configure ... Branch: HEAD Stage Configure failed with status 1 $ It appears to be a "configure" script looking for python; and there is no such. You can have python3 or python2 - but neither package install provides a symlink of just "python". Obviously I could get going by manually adding one, but perhaps other people would benefit from a neater fix. -- Cheers, Jeremy
Jeremy Harris <jgh@wizmail.org> writes:
> Trying to set up a buildfarm animal on a RHEL 8.0 (beta) system,
> the build fails early:
> ...
> It appears to be a "configure" script looking for python; and there is
> no such. You can have python3 or python2 - but neither package install
> provides a symlink of just "python".
Yeah, some distros are starting to act that way, and I suspect it's
causing pain for a lot of people.
Currently we are agnostic about which python version to use, so if you
don't have anything simply named "python", you have to tell configure
what to use by setting the PYTHON environment variable.
In a buildfarm configuration file this would look something like
# env settings to pass to configure. These settings will only be seen by
# configure.
config_env => {
+ PYTHON => "/usr/local/bin/python3",
There's been some preliminary discussion about starting to default to
python3, but given this project's inherent conservatism, I don't expect
that to happen for some years yet. In any case, whenever we do pull
that trigger we'd surely do so only in HEAD not released branches, so
buildfarm owners will need to deal with the case for years more.
regards, tom lane
On 11/24/18 3:49 PM, Tom Lane wrote:
> Jeremy Harris <jgh@wizmail.org> writes:
>> Trying to set up a buildfarm animal on a RHEL 8.0 (beta) system,
>> the build fails early:
>> ...
>> It appears to be a "configure" script looking for python; and there is
>> no such. You can have python3 or python2 - but neither package install
>> provides a symlink of just "python".
> Yeah, some distros are starting to act that way, and I suspect it's
> causing pain for a lot of people.
>
> Currently we are agnostic about which python version to use, so if you
> don't have anything simply named "python", you have to tell configure
> what to use by setting the PYTHON environment variable.
>
> In a buildfarm configuration file this would look something like
>
> # env settings to pass to configure. These settings will only be seen by
> # configure.
> config_env => {
> + PYTHON => "/usr/local/bin/python3",
or just
PYTHON => 'python3',
It's installed by default on RH8, although you will also need
python3-devel installed.
>
> There's been some preliminary discussion about starting to default to
> python3, but given this project's inherent conservatism, I don't expect
> that to happen for some years yet. In any case, whenever we do pull
> that trigger we'd surely do so only in HEAD not released branches, so
> buildfarm owners will need to deal with the case for years more.
>
>
Right.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Hi,
On 2018-11-24 15:49:25 -0500, Tom Lane wrote:
> Jeremy Harris <jgh@wizmail.org> writes:
> > Trying to set up a buildfarm animal on a RHEL 8.0 (beta) system,
> > the build fails early:
> > ...
> > It appears to be a "configure" script looking for python; and there is
> > no such. You can have python3 or python2 - but neither package install
> > provides a symlink of just "python".
>
> Yeah, some distros are starting to act that way, and I suspect it's
> causing pain for a lot of people.
>
> Currently we are agnostic about which python version to use, so if you
> don't have anything simply named "python", you have to tell configure
> what to use by setting the PYTHON environment variable.
>
> In a buildfarm configuration file this would look something like
>
> # env settings to pass to configure. These settings will only be seen by
> # configure.
> config_env => {
> + PYTHON => "/usr/local/bin/python3",
>
> There's been some preliminary discussion about starting to default to
> python3, but given this project's inherent conservatism, I don't expect
> that to happen for some years yet. In any case, whenever we do pull
> that trigger we'd surely do so only in HEAD not released branches, so
> buildfarm owners will need to deal with the case for years more.
Why don't we probe for python2 in addition to python by default? That
ought to make RHEL 8 work, without making the switch just yet.
Greetings,
Andres Freund
Andres Freund <andres@anarazel.de> writes:
> On 2018-11-24 15:49:25 -0500, Tom Lane wrote:
>> There's been some preliminary discussion about starting to default to
>> python3, but given this project's inherent conservatism, I don't expect
>> that to happen for some years yet. In any case, whenever we do pull
>> that trigger we'd surely do so only in HEAD not released branches, so
>> buildfarm owners will need to deal with the case for years more.
> Why don't we probe for python2 in addition to python by default? That
> ought to make RHEL 8 work, without making the switch just yet.
I'm unexcited about that because that *would* be expressing a version
preference --- and one that's on the wrong side of history.
Also, I noticed on a fresh FreeBSD 12.0 installation that what
I've got is
$ ls /usr/bin/pyth*
ls: /usr/bin/pyth*: No such file or directory
$ ls /usr/local/bin/pyth*
/usr/local/bin/python2.7
/usr/local/bin/python2.7-config
/usr/local/bin/python3.6
/usr/local/bin/python3.6-config
/usr/local/bin/python3.6m
/usr/local/bin/python3.6m-config
So there are modern platforms on which "python2" isn't going to make
it work automatically either.
At some point I think what we're going to want to do is to probe for,
in order, $PYTHON, python, python3, python3.7, python3.6, ..., python3.0,
python2, python2.7, python2.6, ..., python2.4 (or whatever our minimum
supported version is). However, I don't think we are ready to put in a
preference towards python3 yet, and it's hard to see how to do something
like this while still being version agnostic.
[ thinks for awhile ... ] Maybe we could do something like this:
1. If $PYTHON exists, use that.
2. If "python" exists, use that.
3. Probe all the numbered python versions suggested above. If
*exactly one* of them exists, use that.
4. Otherwise complain and tell user to resolve uncertainty by
setting $PYTHON.
That's still not quite right, because it'd get confused by
something like symlinking python3 to python3.6, which you'd
kind of wish didn't confuse it. But you could imagine sweating
over this for an hour or two and getting something that generally
did the right thing, and maybe with another hour or two on docs
you could explain it reasonably. I'm unconvinced that it's worth
the trouble though; I think we're better off waiting until we can
go with a straight prefer-the-newest-version rule. With where the
world is right now, it seems to me that making the user specify
which Python to use is arguably a feature not a bug.
regards, tom lane
On 25/11/2018 23:14, Tom Lane wrote: > Andres Freund <andres@anarazel.de> writes: >> On 2018-11-24 15:49:25 -0500, Tom Lane wrote: >>> There's been some preliminary discussion about starting to default to >>> python3, but given this project's inherent conservatism, I don't expect >>> that to happen for some years yet. In any case, whenever we do pull >>> that trigger we'd surely do so only in HEAD not released branches, so >>> buildfarm owners will need to deal with the case for years more. > >> Why don't we probe for python2 in addition to python by default? That >> ought to make RHEL 8 work, without making the switch just yet. > > I'm unexcited about that because that *would* be expressing a version > preference --- and one that's on the wrong side of history. I think it would be appropriate to probe in the order python python3 python2 This would satisfy most scenarios that are valid under PEP 394. > Also, I noticed on a fresh FreeBSD 12.0 installation that what > I've got is > > $ ls /usr/bin/pyth* > ls: /usr/bin/pyth*: No such file or directory > $ ls /usr/local/bin/pyth* > /usr/local/bin/python2.7 > /usr/local/bin/python2.7-config > /usr/local/bin/python3.6 > /usr/local/bin/python3.6-config > /usr/local/bin/python3.6m > /usr/local/bin/python3.6m-config > > So there are modern platforms on which "python2" isn't going to make > it work automatically either. I don't think this is a setup we need to support. You are probably suppose to install a meta package that will provide a "python" or "python3" binary. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Hi, On 2018-11-27 14:14:24 +0100, Peter Eisentraut wrote: > On 25/11/2018 23:14, Tom Lane wrote: > > Andres Freund <andres@anarazel.de> writes: > >> On 2018-11-24 15:49:25 -0500, Tom Lane wrote: > >>> There's been some preliminary discussion about starting to default to > >>> python3, but given this project's inherent conservatism, I don't expect > >>> that to happen for some years yet. In any case, whenever we do pull > >>> that trigger we'd surely do so only in HEAD not released branches, so > >>> buildfarm owners will need to deal with the case for years more. > > > >> Why don't we probe for python2 in addition to python by default? That > >> ought to make RHEL 8 work, without making the switch just yet. > > > > I'm unexcited about that because that *would* be expressing a version > > preference --- and one that's on the wrong side of history. > > I think it would be appropriate to probe in the order > > python python3 python2 > > This would satisfy most scenarios that are valid under PEP 394. ISTM we should first go for python, python2, python3 in all branches, and then have a separate master only commit that changes over the order to prefer python3 over 2. I don't think preferring 3 over 2 would be appropriate for past branches, but it'll surely become more common to run into distros without "python" installed. Greetings, Andres Freund
On 27/11/2018 18:30, Andres Freund wrote: > ISTM we should first go for python, python2, python3 in all branches, > and then have a separate master only commit that changes over the order > to prefer python3 over 2. I don't think preferring 3 over 2 would be > appropriate for past branches, but it'll surely become more common to > run into distros without "python" installed. Thinking about this some more ... Leaving RHEL 8 aside, it would be reasonable to update the search order to "python, python3". That is because the next round of Linux distros is moving to Python 3 as their default, and a fresh install will only provide "python3" and no "python". (You can still install "python"+"python2" separately, at least for a while.) They might change "python" to point to "python3" in the future, but that is currently undecided. There will be a window of time where only "python3" will exist in default installations. (Note that adding "python2" to the search list doesn't affect the above argument, because those installations won't have that either.) Now, about RHEL 8. The reason why RHEL 8 doesn't have "python" is explained here: https://developers.redhat.com/blog/2018/11/27/what-no-python-in-rhel-8-beta/. The summary is: They can't decide what "python" should point to and don't want to be stuck with a decision for 10 years that might end up being incompatible with where the rest of the world moves (via a PEP 394 update). That's something we should consider as well. We need to maintain these decisions for about 5 years, but it's quite plausible that significant changes (or explicit decisions not to change anything) will happen within that timeframe. So, for example, if we were to backpatch the search order "python, python2, python3" to PG11 and then in a few years "python" points to "python3", then we'll end up with an effective search order of "python3, python2, python3" which might be confusing and would create weird inconsistencies across different releases of the same operating system distribution. The fewer things we put in the search list, the fewer chances of weird confusion and unexpected behavior we'll have. My suggestion is to either - Change the probe order to python, python3 in master, or - Don't change anything until PEP 394 is updated. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> My suggestion is to either
> - Change the probe order to python, python3 in master, or
> - Don't change anything until PEP 394 is updated.
It's hard to evaluate the latter unless we have some idea how soon
that update is likely to happen ... have you heard any little birdies
chattering about that lately?
regards, tom lane
On Wed, Nov 28, 2018 at 2:14 AM Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote: > On 25/11/2018 23:14, Tom Lane wrote: > > Also, I noticed on a fresh FreeBSD 12.0 installation that what > > I've got is > > > > $ ls /usr/bin/pyth* > > ls: /usr/bin/pyth*: No such file or directory > > $ ls /usr/local/bin/pyth* > > /usr/local/bin/python2.7 > > /usr/local/bin/python2.7-config > > /usr/local/bin/python3.6 > > /usr/local/bin/python3.6-config > > /usr/local/bin/python3.6m > > /usr/local/bin/python3.6m-config > > > > So there are modern platforms on which "python2" isn't going to make > > it work automatically either. > > I don't think this is a setup we need to support. You are probably > suppose to install a meta package that will provide a "python" or > "python3" binary. Confirmed that if you pkg install python2 and python3 meta-packages on a FreeBSD box you get python2 and python3 commands (symlinks to python2.7, python3.6). -- Thomas Munro http://www.enterprisedb.com
Thomas Munro <thomas.munro@enterprisedb.com> writes:
>> On 25/11/2018 23:14, Tom Lane wrote:
>>> Also, I noticed on a fresh FreeBSD 12.0 installation that what
>>> I've got is [ python2.7 and python3.6 ]
> Confirmed that if you pkg install python2 and python3 meta-packages on
> a FreeBSD box you get python2 and python3 commands (symlinks to
> python2.7, python3.6).
Hm. Retracing my steps, I don't think I explicitly asked for either.
It looks like those got pulled in by installing emacs, which has a
rather monstrous pile of dependencies. But it's weird that those
dependencies aren't phrased as pointing to the meta-packages.
regards, tom lane
On 28/11/2018 22:41, Tom Lane wrote: > Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes: >> My suggestion is to either >> - Change the probe order to python, python3 in master, or >> - Don't change anything until PEP 394 is updated. > > It's hard to evaluate the latter unless we have some idea how soon > that update is likely to happen ... have you heard any little birdies > chattering about that lately? No, I haven't heard anything. I would be slightly surprised if anything changed there before PG12 is released. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 28/11/2018 22:41, Tom Lane wrote:
>> Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
>>> My suggestion is to either
>>> - Change the probe order to python, python3 in master, or
>>> - Don't change anything until PEP 394 is updated.
>> It's hard to evaluate the latter unless we have some idea how soon
>> that update is likely to happen ... have you heard any little birdies
>> chattering about that lately?
> No, I haven't heard anything. I would be slightly surprised if anything
> changed there before PG12 is released.
Yeah, I think your option 2 is effectively the same as "do nothing".
After sleeping on it for awhile, I am liking the idea of probing
python, python3, python2 (while keeping the $PYTHON override of
course). This continues to work as before for any case that
worked before, ie when you either have "python" or used $PYTHON.
It also works for cases where you have only "python3" or only
"python2", which per this thread are becoming more common.
If you have both, then we're stuck with either making an
arbitrary choice or failing. Your proposal at the top is to
arbitrarily choose "python3", which I'm okay with. But if we're
touching this at all, I don't see a good excuse for failing
when the only thing we can find is "python2". The user's intent
seems pretty clear in that case.
Also, I think it'd be fine to back-patch such a change, again
on the grounds that it isn't breaking any case that worked before,
and people are likely to wish to use already-released branches
on platforms that stopped supplying "python".
regards, tom lane
On 29/11/2018 16:34, Tom Lane wrote: > After sleeping on it for awhile, I am liking the idea of probing > python, python3, python2 (while keeping the $PYTHON override of > course). I think this was the option with the most support. Here is a patch. I agree we can backport this. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachment
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 29/11/2018 16:34, Tom Lane wrote:
>> After sleeping on it for awhile, I am liking the idea of probing
>> python, python3, python2 (while keeping the $PYTHON override of
>> course).
> I think this was the option with the most support. Here is a patch.
> I agree we can backport this.
Patch looks fine as far as it goes, but do we need to adjust anything
in installation.sgml or plpython.sgml to explain it?
regards, tom lane
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> I think this was the option with the most support. Here is a patch.
BTW, this is a pre-existing problem not the fault of this patch,
but while we're fooling with the behavior of python lookup would
be a great time to fix it: we should add something like
AC_ARG_VAR([PYTHON], [path to Python executable])
Aside from improving the "configure --help" documentation, this
will prevent insane results from using inapplicable cached
conclusions about what Python we've got. Since more and more
people seem to be using configure cache files, I think we need
to be more careful about declaring relevant variables.
regards, tom lane
On 03/01/2019 18:52, Tom Lane wrote: > Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes: >> On 29/11/2018 16:34, Tom Lane wrote: >>> After sleeping on it for awhile, I am liking the idea of probing >>> python, python3, python2 (while keeping the $PYTHON override of >>> course). > >> I think this was the option with the most support. Here is a patch. >> I agree we can backport this. > > Patch looks fine as far as it goes, but do we need to adjust anything > in installation.sgml or plpython.sgml to explain it? Committed with some documentation updates. I only backpatched to PG10, because before that the handling of absolute paths vs nonabsolute paths is a bit murky and inconsistent, so I didn't want to add more logic and potentially confusing documentation on top of that. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 07/01/2019 00:16, Tom Lane wrote: > BTW, this is a pre-existing problem not the fault of this patch, > but while we're fooling with the behavior of python lookup would > be a great time to fix it: we should add something like > > AC_ARG_VAR([PYTHON], [path to Python executable]) > > Aside from improving the "configure --help" documentation, this > will prevent insane results from using inapplicable cached > conclusions about what Python we've got. Since more and more > people seem to be using configure cache files, I think we need > to be more careful about declaring relevant variables. Patch attached. I went through the environment variables listed in installation.sgml and came up with the following that would affect subsequent cacheable tests: MSGFMT PERL PYTHON TCLSH The following are listed but don't affect any other tests, so I didn't include them: BISON DTRACE DTRACEFLAGS FLEX XML2_CONFIG -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachment
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 07/01/2019 00:16, Tom Lane wrote:
>> BTW, this is a pre-existing problem not the fault of this patch,
>> but while we're fooling with the behavior of python lookup would
>> be a great time to fix it: we should add something like
>> AC_ARG_VAR([PYTHON], [path to Python executable])
> Patch attached.
LGTM. I'm slightly annoyed that configure's help output doesn't list
the variables in alphabetical order, but that seems to be a pre-existing
problem that we likely can't fix.
> The following are listed but don't affect any other tests, so I didn't
> include them:
> BISON
> DTRACE
> DTRACEFLAGS
> FLEX
> XML2_CONFIG
(slightly confused ...) Surely XML2_CONFIG feeds into what we choose for
CPPFLAGS? If that doesn't matter, why not?
regards, tom lane
On 16/01/2019 17:30, Tom Lane wrote: >> The following are listed but don't affect any other tests, so I didn't >> include them: > >> BISON >> DTRACE >> DTRACEFLAGS >> FLEX >> XML2_CONFIG > > (slightly confused ...) Surely XML2_CONFIG feeds into what we choose for > CPPFLAGS? If that doesn't matter, why not? Committed with that addition. (I think I had thought that XML2_CONFIG only affects separate variables like ICU_CFLAGS etc. but that's not the case.) -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
>>> [ patch to search for python, then python3, then python2 ]
> Committed with some documentation updates.
> I only backpatched to PG10, because before that the handling of absolute
> paths vs nonabsolute paths is a bit murky and inconsistent, so I didn't
> want to add more logic and potentially confusing documentation on top of
> that.
So this subject has just intruded itself again, because new buildfarm
member "morepork" is failing in the pre-v10 branches, because ... you
guessed it ... it has "python3" but not "python". This situation is
going to get worse, not better, as time goes on, so I think we really
should back-patch 7291733ac into all active branches.
You were concerned about the fact that b21c569ce had changed
AC_PATH_PROG to PGAC_PATH_PROGS here, but I think we can just ignore
that and make the patch be s/AC_PATH_PROG/AC_PATH_PROGS/, because
that isn't going to change the behavior of that macro w.r.t.
overriding. It was the places that had had AC_CHECK_PROGS that we
were worried about the behavior of, and this place never did have it.
Demonstration that it won't change anything is the very small delta
in the actual configure script in the attached patch for 9.6.
(I've not tested this yet for 9.5 or 9.4, but it appears to apply
cleanly in those branches.)
Thoughts?
regards, tom lane
diff --git a/config/python.m4 b/config/python.m4
index b95c8ed..b8d5b89 100644
--- a/config/python.m4
+++ b/config/python.m4
@@ -6,10 +6,17 @@
# PGAC_PATH_PYTHON
# ----------------
-# Look for Python and set the output variable 'PYTHON'
-# to 'python' if found, empty otherwise.
+# Look for Python and set the output variable 'PYTHON' if found,
+# fail otherwise.
+#
+# As the Python 3 transition happens and PEP 394 isn't updated, we
+# need to cater to systems that don't have unversioned "python" by
+# default. Some systems ship with "python3" by default and perhaps
+# have "python" in an optional package. Some systems only have
+# "python2" and "python3", in which case it's reasonable to prefer the
+# newer version.
AC_DEFUN([PGAC_PATH_PYTHON],
-[AC_PATH_PROG(PYTHON, python)
+[AC_PATH_PROGS(PYTHON, [python python3 python2])
if test x"$PYTHON" = x""; then
AC_MSG_ERROR([Python not found])
fi
diff --git a/configure b/configure
index cfaf11a..48edbf1 100755
--- a/configure
+++ b/configure
@@ -7683,8 +7683,10 @@ fi
fi
if test "$with_python" = yes; then
- # Extract the first word of "python", so it can be a program name with args.
-set dummy python; ac_word=$2
+ for ac_prog in python python3 python2
+do
+ # Extract the first word of "$ac_prog", so it can be a program name with args.
+set dummy $ac_prog; ac_word=$2
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
$as_echo_n "checking for $ac_word... " >&6; }
if ${ac_cv_path_PYTHON+:} false; then :
@@ -7723,6 +7725,9 @@ $as_echo "no" >&6; }
fi
+ test -n "$PYTHON" && break
+done
+
if test x"$PYTHON" = x""; then
as_fn_error $? "Python not found" "$LINENO" 5
fi
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index 481f2ac..cae7973 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -1441,7 +1441,8 @@ su - postgres
<![%standalone-include[the <application>PL/Python</>
documentation]]>
<![%standalone-ignore[<xref linkend="plpython-python23">]]>
- for more information.
+ for more information. If this is not set, the following are probed
+ in this order: <literal>python python3 python2</literal>.
</para>
</listitem>
</varlistentry>
On 2019-09-07 22:18, Tom Lane wrote: > So this subject has just intruded itself again, because new buildfarm > member "morepork" is failing in the pre-v10 branches, because ... you > guessed it ... it has "python3" but not "python". This situation is > going to get worse, not better, as time goes on, so I think we really > should back-patch 7291733ac into all active branches. OK with me. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> On 2019-09-07 22:18, Tom Lane wrote:
>> So this subject has just intruded itself again, because new buildfarm
>> member "morepork" is failing in the pre-v10 branches, because ... you
>> guessed it ... it has "python3" but not "python". This situation is
>> going to get worse, not better, as time goes on, so I think we really
>> should back-patch 7291733ac into all active branches.
> OK with me.
OK, done.
regards, tom lane