Thread: Re: process crash when a plpython function returns unicode

I've moved this thread from pgsql-bugs to pgsql-hackers; here are
the original messages:

http://archives.postgresql.org/pgsql-bugs/2005-06/msg00105.php
http://archives.postgresql.org/pgsql-bugs/2005-06/msg00107.php

As I mentioned in my followup to the bug report, a simple fix would
appear to be to check for a NULL return value from PyObject_Str()
before calling PyString_AsString() at the following location:
   /* Lines 776-77 in plpython.c */   plrv_so = PyObject_Str(plrv);   plrv_sc = PyString_AsString(plrv_so);

I was going to submit a patch, but I don't know enough about the
Python API or how Python and PostgreSQL handle Unicode to know
whether adding that simple check is the appropriate solution (I was
planning to raise an error if PyObject_Str() returned NULL).  Can
anybody think of a better fix?

-- 
Michael Fuhr
http://www.fuhr.org/~mfuhr/

Am Samstag, den 18.06.2005, 08:41 -0600 schrieb Michael Fuhr:
> I've moved this thread from pgsql-bugs to pgsql-hackers; here are
> the original messages:
> 
> http://archives.postgresql.org/pgsql-bugs/2005-06/msg00105.php
> http://archives.postgresql.org/pgsql-bugs/2005-06/msg00107.php
> 
> As I mentioned in my followup to the bug report, a simple fix would
> appear to be to check for a NULL return value from PyObject_Str()
> before calling PyString_AsString() at the following location:
> 
>     /* Lines 776-77 in plpython.c */
>     plrv_so = PyObject_Str(plrv);
>     plrv_sc = PyString_AsString(plrv_so);
> 
> I was going to submit a patch, but I don't know enough about the
> Python API or how Python and PostgreSQL handle Unicode to know
> whether adding that simple check is the appropriate solution (I was
> planning to raise an error if PyObject_Str() returned NULL).  Can
> anybody think of a better fix?

raise error would be a correct solution since this is what
python does in this case:

http://docs.python.org/api/exceptions.html

also in this context it would be helpful
if sys.defaultencoding would be set to
the database encoding so strings get encoded
to utf-8 when postgres works in unicode mode
rather then the default encoding of ascii.
This could avoid most of the PyObject_Str()
exeptions in the first place.

On Sat, Jun 18, 2005 at 05:27:28PM +0200, Tino Wildenhain wrote:
> Am Samstag, den 18.06.2005, 08:41 -0600 schrieb Michael Fuhr:
> > 
> > I was going to submit a patch, but I don't know enough about the
> > Python API or how Python and PostgreSQL handle Unicode to know
> > whether adding that simple check is the appropriate solution (I was
> > planning to raise an error if PyObject_Str() returned NULL).  Can
> > anybody think of a better fix?
> 
> raise error would be a correct solution since this is what
> python does in this case:

I just submitted a patch that checks for NULL and raises an error
via PLy_elog().

> also in this context it would be helpful
> if sys.defaultencoding would be set to
> the database encoding so strings get encoded
> to utf-8 when postgres works in unicode mode
> rather then the default encoding of ascii.
> This could avoid most of the PyObject_Str()
> exeptions in the first place.

I haven't looked at doing that yet and probably won't before feature
freeze.  Gerrit van Dyk has expressed an interest in hacking on
PL/Python (he recently submitted a SETOF patch) so maybe he'll work
on it.

-- 
Michael Fuhr
http://www.fuhr.org/~mfuhr/

On Mon, 2005-06-27 at 08:12 -0600, Michael Fuhr wrote:
> > also in this context it would be helpful
> > if sys.defaultencoding would be set to
> > the database encoding so strings get encoded
> > to utf-8 when postgres works in unicode mode
> > rather then the default encoding of ascii.
> > This could avoid most of the PyObject_Str()
> > exeptions in the first place.
>
> I haven't looked at doing that yet and probably won't before feature
> freeze.  Gerrit van Dyk has expressed an interest in hacking on
> PL/Python (he recently submitted a SETOF patch) so maybe he'll work
> on it.

I have this fixed, for the most part, in PL/Py. What I have done might
be a good starting place for someone who wants to get it fixed in core.

http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/python/be/src/encoding.c

This file makes using PostgreSQL encodings in Python a more friendly
experience by setting up some aliases. (u"óäæ".encode('UNICODE') would
work in 8.0)

Also, to set the default encoding used by Python's Unicode strings:
PyUnicode_SetDefaultEncoding(PyEncoding_FromPgEncoding(GetDatabaseEncoding()))

PyEncoding_FromPgEncoding is defined in encoding.c.

Also, it should be noted that to get the interpreter to read the
function code as a specific encoding, one must use, afaik, the # -*-
encoding: utf-8 -*- magic.
--
Regards, James William Pye