Thread: Pl/Python -- current maintainer?

Pl/Python -- current maintainer?

From
James Robinson
Date:
I'm interested in poking though and taking a shot at getting my feet  
wet with pl/python. I see the file is copyright Andrew Bosma -- is he  
still around perhance? Is anyone currently the 'owner' ?

----
James Robinson
Socialserve.com



Re: Pl/Python -- current maintainer?

From
"Joshua D. Drake"
Date:
James Robinson wrote:
> I'm interested in poking though and taking a shot at getting my feet 
> wet with pl/python. I see the file is copyright Andrew Bosma -- is he 
> still around perhance? Is anyone currently the 'owner' ?
To my knowledge there is no current maintainer of plPython and it 
definitely needs some love.

>
> ----
> James Robinson
> Socialserve.com
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
>       subscribe-nomail command to majordomo@postgresql.org so that your
>       message can get through to the mailing list cleanly
>


-- 
The PostgreSQL Company - Command Prompt, Inc. 1.503.667.4564
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Managed Services, Shared and Dedicated Hosting
Co-Authors: PLphp, PLperl - http://www.commandprompt.com/



Re: Pl/Python -- current maintainer?

From
James Robinson
Date:
I see neilc has hacked on it very recently to reduce memory leaks. I  
take that as both good and bad signs.

We're a [ small ] python shop, and would be most interested in being  
able to simplify our life through doing some things in plpython  
instead of pl/pgsql where appropriate. Keeping our constants and so  
forth in the appropriate python module would make things ever so much  
simpler here and there at the very least.

But we've never hacked on the backend, nor at the C python API level.  
But I see no reason why not to start now -- lurked here for many a  
year. For example, I see that plpython functions cannot be declared  
to return void. That can't be too tough to remedy. Implementing the  
DBI 2.0 API interface to SPI can wait another day.


On Feb 24, 2006, at 11:08 PM, Joshua D. Drake wrote:

> James Robinson wrote:
>> I'm interested in poking though and taking a shot at getting my  
>> feet wet with pl/python. I see the file is copyright Andrew Bosma  
>> -- is he still around perhance? Is anyone currently the 'owner' ?
> To my knowledge there is no current maintainer of plPython and it  
> definitely needs some love.
>
>>
>> ----
>> James Robinson
>> Socialserve.com
>>
>>
>> ---------------------------(end of  
>> broadcast)---------------------------
>> TIP 1: if posting/reading through Usenet, please send an appropriate
>>       subscribe-nomail command to majordomo@postgresql.org so that  
>> your
>>       message can get through to the mailing list cleanly
>>
>
>
> -- 
> The PostgreSQL Company - Command Prompt, Inc. 1.503.667.4564
> PostgreSQL Replication, Consulting, Custom Development, 24x7 support
> Managed Services, Shared and Dedicated Hosting
> Co-Authors: PLphp, PLperl - http://www.commandprompt.com/
>

----
James Robinson
Socialserve.com



Re: Pl/Python -- current maintainer?

From
Tino Wildenhain
Date:
James Robinson schrieb:
> I see neilc has hacked on it very recently to reduce memory leaks. I 
> take that as both good and bad signs.
> 
> We're a [ small ] python shop, and would be most interested in being 
> able to simplify our life through doing some things in plpython  instead
> of pl/pgsql where appropriate. Keeping our constants and so  forth in
> the appropriate python module would make things ever so much  simpler
> here and there at the very least.
> 
> But we've never hacked on the backend, nor at the C python API level. 
> But I see no reason why not to start now -- lurked here for many a 
> year. For example, I see that plpython functions cannot be declared  to
> return void. That can't be too tough to remedy. Implementing the  DBI
> 2.0 API interface to SPI can wait another day.

Also have a look at: http://python.projects.postgresql.org/
it needs some more love too but has high potential.
Maybe it can become next generation pl/pythonu? Would be nice.
And with even more love the restricted python from zope could
be ported so there could be a pl/python again :-)

Ok, just haluzinating ;)

Tino


Re: Pl/Python -- current maintainer?

From
Hannu Krosing
Date:
Ühel kenal päeval, L, 2006-02-25 kell 10:09, kirjutas Tino Wildenhain:
> James Robinson schrieb:
> > I see neilc has hacked on it very recently to reduce memory leaks. I 
> > take that as both good and bad signs.
> > 
> > We're a [ small ] python shop, and would be most interested in being 
> > able to simplify our life through doing some things in plpython  instead
> > of pl/pgsql where appropriate. Keeping our constants and so  forth in
> > the appropriate python module would make things ever so much  simpler
> > here and there at the very least.
> > 
> > But we've never hacked on the backend, nor at the C python API level. 
> > But I see no reason why not to start now -- lurked here for many a 
> > year. For example, I see that plpython functions cannot be declared  to
> > return void. That can't be too tough to remedy. Implementing the  DBI
> > 2.0 API interface to SPI can wait another day.
> 
> Also have a look at: http://python.projects.postgresql.org/
> it needs some more love too but has high potential.

Yes, this one seems to be the pl/python done right :)

But it also suffers a little from trying to do too much at one time, and
so moves a little slow :(

> Maybe it can become next generation pl/pythonu? Would be nice.

I guess that it would need to be at least somewhat backwards compatible
to replace current pl/pythonu.

> And with even more love the restricted python from zope could
> be ported so there could be a pl/python again :-)

That would be nice, but actually not very high on my list of wishes, as
I mostly want to use plpythony as a replacement for writing C funtions,
and we probably will never have "restricted C"

--------------
Hannu




Re: Pl/Python -- current maintainer?

From
James William Pye
Date:
On Sat, Feb 25, 2006 at 10:09:52AM +0100, Tino Wildenhain wrote:
> And with even more love the restricted python from zope could
> be ported so there could be a pl/python again :-)
> 
> Ok, just haluzinating ;)

Not necessarily. ;)

From what I have seen of zope's restricted python, it does, or can, force its
restrictions by checking bytecode. I imagine a simple PL sitting on top of the
untrusted varient that merely implements a custom validator that checks the
bytecode produced by the untrusted PL's validator. The language handler would
remain the same:

<Create untrusted plpy...>

CREATE FUNCTION zope_restrict(oid) RETURNS VOID LANGUAGE python AS
$$
getcode = Postgres.CachedQuery("SELECT probin FROM pg_proc WHERE oid = $1")
getlangval = Postgres.CachedQuery("""SELECT lanvalidator FROM pg_language WHERE    oid = (SELECT prolang FROM pg_proc
WHEREoid = $1)
 
""")

langvaloid = getlangval(self.function.procid)
langval = Postgres.Function(langvaloid)

# produce the function's bytecode for checking
langval(args[0])

# get the function's newly created bytecode
code = getcode(args[0]).next()[0]

#
# XXX: do the zope restriction stuff, raise exception if necessary
#

Postgres.NOTICE(repr(code))
$$;

CREATE TRUSTED LANGUAGE plpyr HANDLER python.handler VALIDATOR zope_restrict;

[This almost works in plpy head, but I think I just found a bug ;]
-- 
Regards, James William Pye


Re: Pl/Python -- current maintainer?

From
Alvaro Herrera
Date:
James William Pye wrote:
> On Sat, Feb 25, 2006 at 10:09:52AM +0100, Tino Wildenhain wrote:
> > And with even more love the restricted python from zope could
> > be ported so there could be a pl/python again :-)
> > 
> > Ok, just haluzinating ;)
> 
> Not necessarily. ;)
> 
> From what I have seen of zope's restricted python, it does, or can, force its
> restrictions by checking bytecode. I imagine a simple PL sitting on top of the
> untrusted varient that merely implements a custom validator that checks the
> bytecode produced by the untrusted PL's validator.

I'm not sure it's an issue now that we have pg_pltemplate, but in older
versions it's possible to create a language without setting a validator.
This would make the validator an unsuitable place for checking the
restrictions.  But the call handler can access the bytecode just the
same, so it's just a matter of moving the checks there, just before the
execution.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support


Re: Pl/Python -- current maintainer?

From
James William Pye
Date:
On Sat, Feb 25, 2006 at 06:36:19PM -0300, Alvaro Herrera wrote:
> I'm not sure it's an issue now that we have pg_pltemplate, but in older
> versions it's possible to create a language without setting a validator.
> This would make the validator an unsuitable place for checking the
> restrictions.

Hrm. I think this would only be an issue in PL/Py is if the user had the ability
to alter probin. The handler will never directly execute code in prosrc; it
relies on a validator to fill in probin.

Whether a regular user could take advantage of this or not, I'm not sure as I
have yet to test it or to give it much thought.
-- 
Regards, James William Pye


Re: Pl/Python -- current maintainer?

From
James William Pye
Date:
On Sat, Feb 25, 2006 at 01:21:34PM -0700, I wrote:
> From what I have seen of zope's restricted python, it does, or can, force its
> restrictions by checking bytecode. I imagine a simple PL sitting on top of the
> untrusted varient that merely implements a custom validator that checks the
> bytecode produced by the untrusted PL's validator. The language handler would
> remain the same:
[ugh, Correcting my assumptions...]

Zope's RestrictedPython is a custom bytecode generator that compiles Python
code specially, as opposed to a bytecode processor that validates against some
rule set as I had thought for some (wishful? ;) reason. The bytecode then needs
to be executed in an special environment that then imposes some specified
restrictions at runtime(I'm not really clear on all the details here as I
am having a very difficult time finding documentation).

This doesn't mean that it couldn't be used. However, it does mean that some
munging of the handler would be required(Something that I desired to avoid).
-- 
Regards, James William Pye


Re: Pl/Python -- current maintainer?

From
Tom Lane
Date:
James William Pye <pgsql@jwp.name> writes:
> On Sat, Feb 25, 2006 at 06:36:19PM -0300, Alvaro Herrera wrote:
>> I'm not sure it's an issue now that we have pg_pltemplate, but in older
>> versions it's possible to create a language without setting a validator.
>> This would make the validator an unsuitable place for checking the
>> restrictions.

> Hrm. I think this would only be an issue in PL/Py is if the user had the ability
> to alter probin. The handler will never directly execute code in prosrc; it
> relies on a validator to fill in probin.

That design is broken on its face, as the system does not guarantee to
call the validator.
        regards, tom lane


Re: Pl/Python -- current maintainer?

From
James William Pye
Date:
On Sun, Feb 26, 2006 at 01:08:52PM -0500, Tom Lane wrote:
> That design is broken on its face, as the system does not guarantee to
> call the validator.

Hrm. Other than language creations that do not specify a validator, at what
times will Postgres not call the validator upon function creation?
-- 
Regards, James William Pye


Re: Pl/Python -- current maintainer?

From
"Michael Paesold"
Date:
James William Pye wrote:

> On Sun, Feb 26, 2006 at 01:08:52PM -0500, Tom Lane wrote:
>> That design is broken on its face, as the system does not guarantee to
>> call the validator.
>
> Hrm. Other than language creations that do not specify a validator, at 
> what
> times will Postgres not call the validator upon function creation?

SET check_function_bodies = off;


Best Regards,
Michael Paesold 




Re: Pl/Python -- current maintainer?

From
Tino Wildenhain
Date:
James William Pye schrieb:
> On Sat, Feb 25, 2006 at 01:21:34PM -0700, I wrote:
> 
>>From what I have seen of zope's restricted python, it does, or can, force its
>>restrictions by checking bytecode. I imagine a simple PL sitting on top of the
>>untrusted varient that merely implements a custom validator that checks the
>>bytecode produced by the untrusted PL's validator. The language handler would
>>remain the same:
> 
> [ugh, Correcting my assumptions...]
> 
> Zope's RestrictedPython is a custom bytecode generator that compiles Python
> code specially, as opposed to a bytecode processor that validates against some
> rule set as I had thought for some (wishful? ;) reason. The bytecode then needs

The key point is: it replaces dangerous elements while it compiles the
bytecode - in theory you could also walk the tree after the python
bytecode compiler (not sure if it even works this way)

for example eval() open() file() import, ... are/can be replaced in this
step.

> to be executed in an special environment that then imposes some specified
> restrictions at runtime(I'm not really clear on all the details here as I
> am having a very difficult time finding documentation).

The special environment is there for the fine grained security only zope
would need in this case. (It protects object attributes individually
while maintaining acquisition and all that stuff)

> This doesn't mean that it couldn't be used. However, it does mean that some
> munging of the handler would be required(Something that I desired to avoid).

You should be able to use most of that technique in the stage where you
create the bytecode - just the step pl/plsql does too.

Regards
Tino


Re: Pl/Python -- current maintainer?

From
James William Pye
Date:
On Sun, Feb 26, 2006 at 07:36:01PM +0100, Michael Paesold wrote:
> SET check_function_bodies = off;

Hrm, thanks for pointing that out. However, this, thankfully, does not appear
to inhibit the validator at all. Rather, the PL's validator is left with the job
to respect it if need be:

jwp=# select version();                                            version

------------------------------------------------------------------------------------
-------------PostgreSQL 8.1.3 on i386-unknown-freebsd6.1, compiled by GCC gcc (GCC) 3.4.4 [FreeB
SD] 20050518
(1 row)

jwp=# set check_function_bodies = off;
SET
jwp=# SHOW check_function_bodies ;check_function_bodies
-----------------------off
(1 row)

jwp=# CREATE FUNCTION test() RETURNS int LANGUAGE python AS 'return 1';
CREATE FUNCTION
jwp=# SELECT prolang, probin FROM pg_proc WHERE proname = 'test';prolang |

probin

---------+--------------------------------------------------------------------------
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
--------------------------------------------------------------------------------  16390 |
c\002\000\000\000\002\000\000\000\001\000\000\000C\000\000\000s\004\000\0
00\000d\001\000S(\002\000\000\000Ni\001\000\000\000(\000\000\000\000(\002\000\000\00
0s\004\000\000\000selfs\004\000\000\000args(\000\000\000\000(\000\000\000\000s\005\0
00\000\00069121s\006\000\000\000test()\002\000\000\000s\002\000\000\000\000\001
(1 row)

jwp=# SELECT test();test
------   1
(1 row)
^D


This is a good thing, IMO. Python primarily does a syntax check so the cited[1]
reasons for the use of check_function_bodies do not seem to apply here.
(Not sure if this is different on 8.0 or earlier.)

[1] http://www.postgresql.org/docs/8.1/static/runtime-config-client.html
-- 
Regards, James William Pye