Thread: Minimum supported version of Python?
Our documentation claims that the minimum Python version for plpython
is 2.3. However, an attempt to build with that on an old Mac yielded
a bunch of failures in the plpython_types regression test, all of the
form
! ERROR: could not import a module for Decimal constructor
! DETAIL: ImportError: No module named decimal
A quick trawl through the buildfarm revealed no evidence that we're
actually testing anything older than Python 2.5, which is why this
wasn't evident awhile ago. It looks like the failing test cases
date back to PG 9.0.
Personally I have no desire to put any effort into fixing this, and
thus suggest that we just change the documentation to specify that 2.5
is the minimum Python version since 9.0. Does anybody else want to
work harder on it, and if so what would you propose doing exactly?
regards, tom lane
On 16 March 2014 11:55, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Our documentation claims that the minimum Python version for plpython > is 2.3. However, an attempt to build with that on an old Mac yielded > a bunch of failures in the plpython_types regression test, all of the > form > ... > Personally I have no desire to put any effort into fixing this, and > thus suggest that we just change the documentation to specify that 2.5 > is the minimum Python version since 9.0. +1 for updating the documentation. 2.5 has been around since 2006 so we are offering a huge range of compatibility as it stands. Versions earlier than 2.5 are probably only of interest to historians at this point. Cheers, BJ
Brendan Jurd <direvus@gmail.com> writes:
> On 16 March 2014 11:55, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Our documentation claims that the minimum Python version for plpython
>> is 2.3. However, an attempt to build with that on an old Mac yielded
>> a bunch of failures in the plpython_types regression test, all of the
>> form
>> ...
>> Personally I have no desire to put any effort into fixing this, and
>> thus suggest that we just change the documentation to specify that 2.5
>> is the minimum Python version since 9.0.
> +1 for updating the documentation. 2.5 has been around since 2006 so
> we are offering a huge range of compatibility as it stands. Versions
> earlier than 2.5 are probably only of interest to historians at this
> point.
Yeah, the lack of complaints from the field suggests strongly that
nobody cares about this, so I'll just go change the docs.
regards, tom lane
On Sat, 2014-03-15 at 20:55 -0400, Tom Lane wrote: > Our documentation claims that the minimum Python version for plpython > is 2.3. However, an attempt to build with that on an old Mac yielded > a bunch of failures in the plpython_types regression test, It has frequently been the case that the last supported version does not fully pass the regression test, because of the overhead of maintaining variant files. The last supported version is the one that compiles and works. You will note that 2.2 no longer compiles. (It also failed the regression tests for a while before it started not compiling.) Typically, versions fall out of support because we add new functionality that the old Python versions cannot support anymore. > all of the form > > ! ERROR: could not import a module for Decimal constructor > ! DETAIL: ImportError: No module named decimal You can make this work by manually installing the decimal module (because it was not part of the core in Python 2.3). Otherwise, this test result legitimately alerts you that some feature is not fully working and that you need to adjust your installation. > A quick trawl through the buildfarm revealed no evidence that we're > actually testing anything older than Python 2.5, which is why this > wasn't evident awhile ago. It was known when the decimal feature was committed. I regularly test all supported Python versions. > It looks like the failing test cases date back to PG 9.0. This feature is new in 9.4, so that can't be quite right. We have added and dropped supported Python versions in most major releases over time, so there will likely be different results and different considerations in each major release. Certainly, all Python versions listed as supported in each major PG release actually worked at the time of release. > Personally I have no desire to put any effort into fixing this, and > thus suggest that we just change the documentation to specify that 2.5 > is the minimum Python version since 9.0. Does anybody else want to > work harder on it, and if so what would you propose doing exactly? It is debatable whether 2.3 should still be considered supported in 9.4. If we do drop it, then the supported version should be 2.4. Personally, I think everything is fine as it is.
On Sun, 2014-03-16 at 15:56 +1100, Brendan Jurd wrote: > Versions > earlier than 2.5 are probably only of interest to historians at this > point. and users of RHEL 5
Peter Eisentraut <peter_e@gmx.net> writes:
> On Sat, 2014-03-15 at 20:55 -0400, Tom Lane wrote:
>> Our documentation claims that the minimum Python version for plpython
>> is 2.3. However, an attempt to build with that on an old Mac yielded
>> a bunch of failures in the plpython_types regression test,
>> Personally I have no desire to put any effort into fixing this, and
>> thus suggest that we just change the documentation to specify that 2.5
>> is the minimum Python version since 9.0. Does anybody else want to
>> work harder on it, and if so what would you propose doing exactly?
> It is debatable whether 2.3 should still be considered supported in 9.4.
> If we do drop it, then the supported version should be 2.4. Personally,
> I think everything is fine as it is.
Well, if you want to consider python 2.3 as supported, I have a buildfarm
machine I am about to put online that has 2.3 on it. If I spin it up with
python enabled, I expect you to see to it that it starts passing. If you
won't do that, I'm going to change the documentation.
As for 2.4 vs 2.5, I don't have a lot of faith that we're really
supporting anything that's not represented in the buildfarm...
regards, tom lane
On Sun, 2014-03-16 at 22:34 -0400, Tom Lane wrote: > Well, if you want to consider python 2.3 as supported, I have a buildfarm > machine I am about to put online that has 2.3 on it. If I spin it up with > python enabled, I expect you to see to it that it starts passing. If you > won't do that, I'm going to change the documentation. As I said, according to my testing, 2.3 is supported. If your experience is different, then please submit a reproducible bug report. > As for 2.4 vs 2.5, I don't have a lot of faith that we're really > supporting anything that's not represented in the buildfarm... There are many other features that the build farm doesn't test and that I don't have a lot of faith in, but I'm not proposing to remove those. I don't control what the build farm tests, I only control my own work.
On 03/17/2014 07:31 PM, Peter Eisentraut wrote: > > On Sun, 2014-03-16 at 22:34 -0400, Tom Lane wrote: >> Well, if you want to consider python 2.3 as supported, I have a buildfarm >> machine I am about to put online that has 2.3 on it. If I spin it up with >> python enabled, I expect you to see to it that it starts passing. If you >> won't do that, I'm going to change the documentation. > > As I said, according to my testing, 2.3 is supported. If your > experience is different, then please submit a reproducible bug report. > >> As for 2.4 vs 2.5, I don't have a lot of faith that we're really >> supporting anything that's not represented in the buildfarm... > > There are many other features that the build farm doesn't test and that > I don't have a lot of faith in, but I'm not proposing to remove those. > I don't control what the build farm tests, I only control my own work. We shouldn't be supporting anything the community doesn't support. Python 2.3 is dead. We shouldn't actively support it nor suggest that we could or should via the docs. There is certainly an argument for Python 2.4 (due to CentOS/RHEL) but other than that... really? JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc Political Correctness is for cowards.
Peter Eisentraut-2 wrote > On Sat, 2014-03-15 at 20:55 -0400, Tom Lane wrote: >> Our documentation claims that the minimum Python version for plpython >> is 2.3. However, an attempt to build with that on an old Mac yielded >> a bunch of failures in the plpython_types regression test, > > It has frequently been the case that the last supported version does not > fully pass the regression test, because of the overhead of maintaining > variant files. The last supported version is the one that compiles and > works. You will note that 2.2 no longer compiles. (It also failed the > regression tests for a while before it started not compiling.) > Typically, versions fall out of support because we add new functionality > that the old Python versions cannot support anymore. > >> all of the form >> >> ! ERROR: could not import a module for Decimal constructor >> ! DETAIL: ImportError: No module named decimal > > You can make this work by manually installing the decimal module > (because it was not part of the core in Python 2.3). Otherwise, this > test result legitimately alerts you that some feature is not fully > working and that you need to adjust your installation. It would seem a single error (or warning, depending) like "Missing Dependency" at the start and skipping all tests dependent on the error would be cleaner. So that if someone goes an blindly starts running a clean 2.3 regression test they are not drowned in a sea of errors and no clear documentation as to why and how to resolve. Now, odds are the assumption that Decimal is present is spread through the code-base so maybe something less invasive? David J. -- View this message in context: http://postgresql.1045698.n5.nabble.com/Minimum-supported-version-of-Python-tp5796175p5796512.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.
On 03/17/2014 07:31 PM, Peter Eisentraut wrote:
>> As I said, according to my testing, 2.3 is supported. If your
>> experience is different, then please submit a reproducible bug report.
It doesn't pass the regression tests. Do you need more of a bug report
than that?
>> There are many other features that the build farm doesn't test and that
>> I don't have a lot of faith in, but I'm not proposing to remove those.
I'm not proposing to remove any code either. What I am proposing is that
we should not describe Python 2.3 as "supported" if our regression tests
don't support it. The most that's going to get for us is questions and
bug reports.
"Joshua D. Drake" <jd@commandprompt.com> writes:
> Python 2.3 is dead. We shouldn't actively support it nor suggest that we
> could or should via the docs.
The lack of field complaints about the regression test issue lends
considerable weight to the theory that nobody cares about 2.3 anymore.
Given that, I'm fully on board with the idea that we should not invest
effort in making the regression tests pass with it. But ... if we aren't
willing to put in that effort, then we're not supporting it. QED.
regards, tom lane
Joshua D. Drake wrote > On 03/17/2014 07:31 PM, Peter Eisentraut wrote: >> >> On Sun, 2014-03-16 at 22:34 -0400, Tom Lane wrote: >>> Well, if you want to consider python 2.3 as supported, I have a >>> buildfarm >>> machine I am about to put online that has 2.3 on it. If I spin it up >>> with >>> python enabled, I expect you to see to it that it starts passing. If >>> you >>> won't do that, I'm going to change the documentation. >> >> As I said, according to my testing, 2.3 is supported. If your >> experience is different, then please submit a reproducible bug report. >> >>> As for 2.4 vs 2.5, I don't have a lot of faith that we're really >>> supporting anything that's not represented in the buildfarm... >> >> There are many other features that the build farm doesn't test and that >> I don't have a lot of faith in, but I'm not proposing to remove those. >> I don't control what the build farm tests, I only control my own work. > > We shouldn't be supporting anything the community doesn't support. > Python 2.3 is dead. We shouldn't actively support it nor suggest that we > could or should via the docs. > > There is certainly an argument for Python 2.4 (due to CentOS/RHEL) but > other than that... really? > > JD +1 generally though it ignores the bigger question which is how are we actually defining and proving support; and also the whole we support 8.4 (for a little longer now) so what related technology should we be either supporting or at least saying "ya know, this was working just fine 4 years ago if you use exactly these versions, so use those versions." A "last know working version" in the plpythonu section wouldn't hurt since actively running all the possible combinations for the 5 year support cycle doesn't make sense. Is Peter /the/ final arbiter of fact in this area or are there some public buildfarm animals out there that have been designated to fulfill this role? Peter claims to be doing the compiling and testing for these, which is great and would be better if people like Tom actually were aware of that fact, but comments suggest that such testing is currently done "offline". To Peter's comment - reporting which version combinations are known to compile is good but if a regression test exists then it should either pass on a clean version or it should be documented (ideally right in the tests themselves) what additional configuration is required (though maybe moving that back to configure would be sufficient for our needs). If there are no coverage tests we are not asserting any proof anyway so if a release compiles and passes regression tests it should be noted as such. Once the regressions start failing without any known fix it's hard to understand how we could say such a combination is valid/supported anymore... Supported - we will endeavor to fix bugs. add the test case and then fix the problem. Compiles/Tests-OK - we know it worked at one point but if we discover a bug we're more than likely to just say that the version is no longer supported and known to have bugs. Add the test case that causes regression to fail and move on. David J. -- View this message in context: http://postgresql.1045698.n5.nabble.com/Minimum-supported-version-of-Python-tp5796175p5796516.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.
On 03/17/2014 10:31 PM, Peter Eisentraut wrote: > On Sun, 2014-03-16 at 22:34 -0400, Tom Lane wrote: >> As for 2.4 vs 2.5, I don't have a lot of faith that we're really >> supporting anything that's not represented in the buildfarm... > There are many other features that the build farm doesn't test and that > I don't have a lot of faith in, but I'm not proposing to remove those. > I don't control what the build farm tests, I only control my own work. > > Actually, if you run a buildfarm animal you have considerable control over what it tests. It's a very loosely coupled distributed system. The whole basis on which it was constructed was that people who were interested in certain combinations of hardware and infrastructure would create animals to test those combinations. If that's too much trouble for people, I have often offered to set up an animal for people if they give me ssh access. If that's too much trouble for people I tend to conclude that they aren't actually all that interested. cheers andrew
On 3/17/14, 10:55 PM, Tom Lane wrote: > It doesn't pass the regression tests. Do you need more of a bug report > than that? It does pass the tests for me and others. If you are seeing something different, then we need to see some details, like what platform, what version, what Python version, how installed, what PostgreSQL version, how installed, actual diffs, etc.
On 3/17/14, 10:47 PM, Joshua D. Drake wrote: > We shouldn't be supporting anything the community doesn't support. > Python 2.3 is dead. We shouldn't actively support it nor suggest that we > could or should via the docs. The information that is available to me about this issue is lacking details, but as far as I can tell, we are partially talking about backbranches, which were released at a time when Python 2.3 and 2.4 were not unreasonable targets. We're not going to desupport build dependencies in the middle of a stable release branch unless we have a solid reason.
On 3/18/14, 11:22 AM, Andrew Dunstan wrote: > Actually, if you run a buildfarm animal you have considerable control > over what it tests. I appreciate that. My problem here isn't time or ideas or coding, but lack of hardware resources. If I had hardware, I could set up tests for every build dependency under the sun.
Peter Eisentraut-2 wrote > On 3/18/14, 11:22 AM, Andrew Dunstan wrote: >> Actually, if you run a buildfarm animal you have considerable control >> over what it tests. > > I appreciate that. My problem here isn't time or ideas or coding, but > lack of hardware resources. If I had hardware, I could set up tests for > every build dependency under the sun. I don't imagine there is enough churn in this area that having a constantly testing buildfarm animal is a strong need; but a wiki page dedicated to "Python Support in PostgreSQL" where we can publicly and officially release testing results and commentary would be an improvement. As it sounds like the only caveat to supporting 2.3 is that we don't technically (or do we - is Decimal mandatory?) support an un-modified core installation but require that at one add-on module be installed. Assuming that clears up all the errors Tom is seeing then saying that plpythonu works with a slightly modified 2.3 on all current releases of PostgreSQL isn't a stretch nor does it commit us to fixing bugs in the unlikely event that any are discovered in two extremely stable environments. -- View this message in context: http://postgresql.1045698.n5.nabble.com/Minimum-supported-version-of-Python-tp5796175p5796615.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.
Peter Eisentraut <peter_e@gmx.net> writes:
> On 3/17/14, 10:55 PM, Tom Lane wrote:
>> It doesn't pass the regression tests. Do you need more of a bug report
>> than that?
> It does pass the tests for me and others. If you are seeing something
> different, then we need to see some details, like what platform, what
> version, what Python version, how installed, what PostgreSQL version,
> how installed, actual diffs, etc.
After further experimentation, I've found that 2.3 does pass the regression
tests if one installs the separately-available cdecimal module. So my
complaint reduces to the fact that our "Requirements" documentation
doesn't mention the need for this. Do you have an objection to adding
something there?
regards, tom lane
I wrote: > Peter Eisentraut <peter_e@gmx.net> writes: >> It does pass the tests for me and others. If you are seeing something >> different, then we need to see some details, like what platform, what >> version, what Python version, how installed, what PostgreSQL version, >> how installed, actual diffs, etc. > After further experimentation, I've found that 2.3 does pass the regression > tests if one installs the separately-available cdecimal module. Well ... it passes in C locale, anyway. 9.1 appears to have a problem if using UTF8 encoding: http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prairiedog&dt=2014-03-19%2017%3A00%3A48 I've had a hard time getting details, since Apple didn't supply debug symbols for their Python build, but the segfault is definitely happening down inside PyRun_String() called from plpython_validator(). If you just fire up a fresh session and execute the troublesome CREATE FUNCTION, it's fine; which says to me that some previous operation in the plpython_trigger test script tromped on data structures private to Python. Our other branches pass on the identical installation. Now, 9.0 didn't even have a validator, which might just mean that it's failing to trip over a memory clobber that happened anyway. And 9.2 is so completely refactored that it's hard to tell whether it incorporates a fix for a memory clobber compared to 9.1; or maybe it's just accidentally avoiding the crash symptom, too. But there's something rotten in the state of Denmark. regards, tom lane
I wrote: > Well ... [2.3] passes in C locale, anyway. 9.1 appears to have a problem if > using UTF8 encoding: > http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prairiedog&dt=2014-03-19%2017%3A00%3A48 Some bisection tests say that it started failing after commit 17dee323e7ff67863582f279e415a8228e0b99cd. No idea why, atm. regards, tom lane
Hi, Robert pointed this out to me, thanks. On 2014-03-24 00:54:31 -0400, Tom Lane wrote: > I wrote: > > Well ... [2.3] passes in C locale, anyway. 9.1 appears to have a problem if > > using UTF8 encoding: > > http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prairiedog&dt=2014-03-19%2017%3A00%3A48 > > Some bisection tests say that it started failing after commit > 17dee323e7ff67863582f279e415a8228e0b99cd. No idea why, atm. I can't really see why that'd would cause any such failure either. One thing that I'd try is is just disabling the caching, to rule out problems around that. Do I understand correctly that you have a machine where you can reproduce this on? And it can just be reproduced on OSX with python 2.3? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Andres Freund <andres@2ndquadrant.com> writes:
> On 2014-03-24 00:54:31 -0400, Tom Lane wrote:
>> Some bisection tests say that it started failing after commit
>> 17dee323e7ff67863582f279e415a8228e0b99cd. No idea why, atm.
> I can't really see why that'd would cause any such failure either. One
> thing that I'd try is is just disabling the caching, to rule out
> problems around that.
> Do I understand correctly that you have a machine where you can
> reproduce this on? And it can just be reproduced on OSX with python 2.3?
It's 100% reproducible on prairiedog, which is a Mac Cube running Tiger,
and using the version of python that came with Tiger (2.3.5 :-(). It
seems not impossible that we're looking at a long-since-fixed python
bug, but the fact that that specific commit made it appear or disappear
is troubling.
One thing I was wondering about, not being a python user: why is it okay
to use non-unique names for the python procedures? That is,
PLy_procedure_munge_source() does a "def" with a procedure name
constructed thus:
rv = snprintf(procName, sizeof(procName), "__plpython_procedure_%s_%u",
NameStr(procStruct->proname), fn_oid);
while at the plpython level we're of the opinion that trigger functions
applied to different relations need separate cache entries. How can it
be OK for separate cache entries to reference the same "def" name?
I tried modifying the code like this:
rv = snprintf(procName, sizeof(procName), "__plpython_%sprocedure_%s_%u_%u", is_trigger ?
"trigger_": "", NameStr(procStruct->proname), fn_oid, fn_rel);
and that didn't make this crash go away, but it sure looks like the
existing code could cause problems for somebody.
regards, tom lane
On 2014-03-25 15:08:16 -0400, Tom Lane wrote: > Andres Freund <andres@2ndquadrant.com> writes: > > On 2014-03-24 00:54:31 -0400, Tom Lane wrote: > >> Some bisection tests say that it started failing after commit > >> 17dee323e7ff67863582f279e415a8228e0b99cd. No idea why, atm. > > > I can't really see why that'd would cause any such failure either. One > > thing that I'd try is is just disabling the caching, to rule out > > problems around that. > > Do I understand correctly that you have a machine where you can > > reproduce this on? And it can just be reproduced on OSX with python 2.3? > > It's 100% reproducible on prairiedog, which is a Mac Cube running Tiger, > and using the version of python that came with Tiger (2.3.5 :-(). It > seems not impossible that we're looking at a long-since-fixed python > bug, but the fact that that specific commit made it appear or disappear > is troubling. Yea, agreed. If there's a refcounting bug inside python somewhere (which is easy to trigger in python's C interface), it could be excerbated by that change, since it frees/compiles functions more frequently. But I'd very much like more evidence of this... > One thing I was wondering about, not being a python user: why is it okay > to use non-unique names for the python procedures? That is, > PLy_procedure_munge_source() does a "def" with a procedure name > constructed thus: That should be fine. The name isn't really meaningful for anything but debugging. It's perfectly fine to overwrite functions in a namespace and call them via reference in python. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Andres Freund <andres@2ndquadrant.com> writes:
> On 2014-03-25 15:08:16 -0400, Tom Lane wrote:
>> It's 100% reproducible on prairiedog, which is a Mac Cube running Tiger,
>> and using the version of python that came with Tiger (2.3.5 :-(). It
>> seems not impossible that we're looking at a long-since-fixed python
>> bug, but the fact that that specific commit made it appear or disappear
>> is troubling.
> Yea, agreed.
> If there's a refcounting bug inside python somewhere (which is easy to
> trigger in python's C interface), it could be excerbated by that change,
> since it frees/compiles functions more frequently. But I'd very much
> like more evidence of this...
I think it's not a refcount issue, or at least not solely that. As best
I can tell, there's a stack clobber involved, because gdb can't make sense
of the stack after the exception hits. I've been trying to localize it
more closely, but it's slow going because Apple's copy of python doesn't
include debug symbols.
regards, tom lane
I wrote:
> Andres Freund <andres@2ndquadrant.com> writes:
>> If there's a refcounting bug inside python somewhere (which is easy to
>> trigger in python's C interface), it could be excerbated by that change,
>> since it frees/compiles functions more frequently. But I'd very much
>> like more evidence of this...
> I think it's not a refcount issue, or at least not solely that. As best
> I can tell, there's a stack clobber involved, because gdb can't make sense
> of the stack after the exception hits. I've been trying to localize it
> more closely, but it's slow going because Apple's copy of python doesn't
> include debug symbols.
Fortunately, Apple still has the source code for that package archived at
www.opensource.apple.com, so after a bit of hair-pulling I was able to
build a version with debug symbols. And (may I have the envelope please)
you're right: it is a refcount issue. Take a look at what
PLy_modify_tuple does with plntup, and note that the "TD[\"new\"] is not a
dictionary" error is intentionally triggered by the plpython_trigger test.
So we have a prematurely-freed dictionary item apparently available for
recycling even though it's still part of the calling function's parsetree.
It's still like that in HEAD, too.
Will fix it shortly. I wonder though if there are any more thinkos like
this one :-(
BTW, isn't plstr totally dead code?
regards, tom lane
On Tue, 2014-03-18 at 18:37 -0400, Tom Lane wrote: > After further experimentation, I've found that 2.3 does pass the > regression > tests if one installs the separately-available cdecimal module. So my > complaint reduces to the fact that our "Requirements" documentation > doesn't mention the need for this. Do you have an objection to adding > something there? You backpatched this change, but that can't be right, because the feature that requires the cdecimal module was added in 9.4 (7919398bac8bacd75ec5d763ce8b15ffaaa3e071).
Peter Eisentraut <peter_e@gmx.net> writes:
> You backpatched this change, but that can't be right, because the
> feature that requires the cdecimal module was added in 9.4
> (7919398bac8bacd75ec5d763ce8b15ffaaa3e071).
Ah. I saw that the failing tests were quite old, but did not realize
that we'd only recently added this way for them to fail. Will adjust.
regards, tom lane