Thread: Re: Iterating generator from C (PostgreSQL's pl/python RETUN

Sorry for cross-posting, but this IS a cross-platform issue.

Christian Tismer tismer at stackless.com  wrote:
> Sven Suursoho wrote:
>
> >>>  Is there any way to rewrite following program to handle returned
> >>> generator  without hitting this bug?
>
> The only way I can think of getting around this is to make
> sure that there is always a running python frame.
> This would be possible if you can control how the
> extension is called.

What would be the easiest way to hold a "always running" python frame ?

The actual calls to iterator come through a pl/python framework, which can hold some
state - like some inter-call dictionaries - so keeping also a simple outer frame
there may be possible.

What danger (apart from general uglyness) may lurk there in keeping this frame ?

> > Unfortunately, this is not an option because I can't control used
> > environment: I'm trying to improve PostgreSQL's stored procedure
> > language PL/Python and this software can be used everywhere.
>
> Then there is no other way than to report the bug, get the
> fix back-ported and nagging the PL/Python maintainers
> to update things after the fix.

Unfortunately there is no 'after the fix', as the fix must happen outside
postgresql/plpython development and should also run on oldish systems.

The reason we want to get some workaround for that bug is the need
to overcome resistance from core postgreSQL developers to inclusion of our
plpython enchancements to postgreSQLs bundled plpython due to one specific use
of our generic enchancement (using a built-in generator, usually a function with yield)
on buggy RedHat's bundled plpython.so causing crashes.

Also, PL/Python has been in minimal maintenance mode for many years, with
basically only immediate bugfixing going on.

We at Skype (that is currently Sven Suursoho) are the first ones doing
active development on it after Andrew Bosma dropped development many years
ago once he got just the very basic stuff working.

> Also a test should be added which is probably missing since a while.

Test where ? In python.so build process, so that RedHat will spot it and
fix their RPM builds ?

As for testing in actual pl/python build environment, we had objections from
leading postgresql Tom Lane that even if we do test it at build time,
a determined DBA may substitute a buggy python.so later and still crash her DB instance.

Do you have any ideas, how to test for buggy asserts in python runtime
environment without actually triggering the assert ?

Then we could introduce some (uglish) special handling for generators in
pl/pythons iterator.

> I'd put a warning somewhere that generators are broken in
> debug mode, file an issue as well, but avoid trying to hack
> around this. It would make the bug even more resistent :-)

We have been trying to advocate such approach, but so far with modest results :(

--
----------------
Hannu Krosing
Database Architect
Skype Technologies OÜ
Akadeemia tee 21 F, Tallinn, 12618, Estonia

Skype me:  callto:hkrosing
Get Skype for free:  http://www.skype.com

Hannu Krosing <hannu@skype.net> writes:
>> Sven Suursoho wrote:
>>> As for testing in actual pl/python build environment, we had objections from
>>> leading postgresql Tom Lane that even if we do test it at build time,
>>> a determined DBA may substitute a buggy python.so later and still crash her DB instance.

The above is a straw-man depiction of my point.  What I said was that just
because python is up-to-date on the system where plpython is *compiled*
does not mean it'll be up-to-date on the system where plpython is *used*.
With the increasing popularity of prebuilt packages (rpm, deb, etc),
I think it's folly to use a build-time check for this.

            regards, tom lane

Ühel kenal päeval, E, 2006-05-15 kell 17:21, kirjutas Tom Lane:
> Hannu Krosing <hannu@skype.net> writes:
> >> Sven Suursoho wrote:
> >>> As for testing in actual pl/python build environment, we had objections from
> >>> leading postgresql Tom Lane that even if we do test it at build time,
> >>> a determined DBA may substitute a buggy python.so later and still crash her DB instance.
>
> The above is a straw-man depiction of my point.

Sure ;)

> What I said was that just
> because python is up-to-date on the system where plpython is *compiled*
> does not mean it'll be up-to-date on the system where plpython is *used*.

Would running an external program at pl init time and testing for its
crash status be a good broken lib test for plpython ?

> With the increasing popularity of prebuilt packages (rpm, deb, etc),
> I think it's folly to use a build-time check for this.

I guess most packaging systems can require some minimal version of
dependencies.

And in general newer versions are less buggy than old ones.

So i guess that some combination of build-time/run-time tests,
documentation and packager education should take care of 99% of
concerns ?

Hopefully we can just ignore the "determined DBA" failure mode and move
forward with including the patch. Or do you think that trying to hack in
the extra python frame to all/some builds to ward of potentially broken
libs would still be something to go for ?

--
----------------
Hannu Krosing
Database Architect
Skype Technologies OÜ
Akadeemia tee 21 F, Tallinn, 12618, Estonia

Skype me:  callto:hkrosing
Get Skype for free:  http://www.skype.com

NOTICE: This communication contains privileged or other confidential
information. If you have received it in error, please advise the sender
by reply email and immediately delete the message and any attachments
without copying or disclosing the contents.