Thread: [PATCH] plpythonu datatype conversion improvements
<font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:11pt">Patch for plpythonu<br /><br /> Primary
motivationof the attached patch is to support handling bytea conversion allowing for embedded nulls, which in turn
allowsfor supporting the marshal module.<br /><br /> Secondary motivation is slightly improved performance for
conversionroutines of basic datatypes that have simple mappings between postgres/python.<br /><br /> Primary design is
tochange the conversion routines from being based on cstrings to datums, eg:<br /> PLyBool_FromString(const char *)
=>PLyBool_FromBool(PLyDatumToOb, Datum);<br /><br /> Thanks, <br /> Caleb<br /><br /> -----<br /><br /> Index:
plpython.c<br/> ===================================================================<br /> RCS file:
/projects/cvsroot/pgsql/src/pl/plpython/plpython.c,v<br/> retrieving revision 1.120<br /> diff -c -r1.120 plpython.c<br
/>*** plpython.c 3 Apr 2009 16:59:42 -0000 1.120<br /> --- plpython.c 26 May 2009 22:58:52 -0000<br />
***************<br/> *** 78,84 ****<br /> * objects.<br /> */<br /> <br /> ! typedef PyObject
*(*PLyDatumToObFunc)(const char *);<br /> <br /> typedef struct PLyDatumToOb<br /> {<br /> --- 78,85 ----<br />
*objects.<br /> */<br /> <br /> ! struct PLyDatumToOb;<br /> ! typedef PyObject *(*PLyDatumToObFunc) (struct
PLyDatumToOb*,Datum);<br /> <br /> typedef struct PLyDatumToOb<br /> {<br /> ***************<br /> *** 104,111
****<br/> --- 105,120 ----<br /> /* convert PyObject to a Postgresql Datum or tuple.<br /> * output from Python<br
/> */<br /> + <br /> + struct PLyObToDatum;<br /> + struct PLyProcedure;<br /> + typedef Datum (*PLyObToDatumFunc)
(structPLyProcedure*, <br /> + struct PLyObToDatum*, <br /> +
PyObject*, bool *isnull);<br /> + <br /> typedef struct PLyObToDatum<br /> {<br
/>+ PLyObToDatumFunc func;<br /> FmgrInfo typfunc; /* The type's input function */<br /> Oid
typoid; /* The OID of the type */<br /> Oid typioparam;<br /> ***************<br
/>*** 255,270 ****<br /> static void PLy_input_tuple_funcs(PLyTypeInfo *, TupleDesc);<br /> <br /> /* conversion
functions*/<br /> static PyObject *PLyDict_FromTuple(PLyTypeInfo *, HeapTuple, TupleDesc);<br /> ! static PyObject
*PLyBool_FromString(constchar *);<br /> ! static PyObject *PLyFloat_FromString(const char *);<br /> ! static PyObject
*PLyInt_FromString(constchar *);<br /> ! static PyObject *PLyLong_FromString(const char *);<br /> ! static PyObject
*PLyString_FromString(constchar *);<br /> ! <br /> ! static HeapTuple PLyMapping_ToTuple(PLyTypeInfo *, PyObject *);<br
/>! static HeapTuple PLySequence_ToTuple(PLyTypeInfo *, PyObject *);<br /> ! static HeapTuple
PLyObject_ToTuple(PLyTypeInfo*, PyObject *);<br /> <br /> /*<br /> * Currently active plpython function<br />
---264,295 ----<br /> static void PLy_input_tuple_funcs(PLyTypeInfo *, TupleDesc);<br /> <br /> /* conversion
functions*/<br /> + static PyObject *PLyBool_FromBool(PLyDatumToOb *arg, Datum d);<br /> + static PyObject
*PLyFloat_FromFloat4(PLyDatumToOb*arg, Datum d);<br /> + static PyObject *PLyFloat_FromFloat8(PLyDatumToOb *arg, Datum
d);<br/> + static PyObject *PLyFloat_FromNumeric(PLyDatumToOb *arg, Datum d);<br /> + static PyObject
*PLyInt_FromInt16(PLyDatumToOb*arg, Datum d);<br /> + static PyObject *PLyInt_FromInt32(PLyDatumToOb *arg, Datum d);<br
/>+ static PyObject *PLyLong_FromInt64(PLyDatumToOb *arg, Datum d);<br /> + static PyObject
*PLyString_FromText(PLyDatumToOb*arg, Datum d);<br /> + static PyObject *PLyString_FromDatum(PLyDatumToOb *arg, Datum
d);<br/> + <br /> static PyObject *PLyDict_FromTuple(PLyTypeInfo *, HeapTuple, TupleDesc);<br /> ! <br /> ! static
DatumPLyObject_ToVoid(PLyProcedure *, PLyObToDatum *, <br /> ! PyObject *, bool
*isnull);<br/> ! static Datum PLyObject_ToBool(PLyProcedure *, PLyObToDatum *, <br /> !
PyObject*, bool *isnull);<br /> ! static Datum PLyObject_ToBytea(PLyProcedure *,
PLyObToDatum*, <br /> ! PyObject *, bool *isnull);<br /> ! static Datum
PLyObject_ToText(PLyProcedure*, PLyObToDatum *, <br /> ! PyObject *, bool *isnull);<br />
!static Datum PLyObject_ToDatum(PLyProcedure *, PLyObToDatum *, <br /> ! PyObject *,
bool*isnull);<br /> ! <br /> ! static HeapTuple PLyMapping_ToTuple(PLyProcedure *, PyObject *);<br /> ! static
HeapTuplePLySequence_ToTuple(PLyProcedure *, PyObject *);<br /> ! static HeapTuple PLyObject_ToTuple(PLyProcedure *,
PyObject*);<br /> <br /> /*<br /> * Currently active plpython function<br /> ***************<br /> *** 507,514
****<br/> <br /> for (i = 0; i < natts; i++)<br /> {<br /> - char *src;<br
/>- <br /> platt = PyList_GetItem(plkeys, i);<br /> if (!PyString_Check(platt))<br />
ereport(ERROR,<br/> --- 532,537 ----<br /> ***************<br /> *** 533,564 ****<br />
modvalues[i]= (Datum) 0;<br /> modnulls[i] = 'n';<br /> }<br /> !
elseif (plval != Py_None)<br /> {<br /> ! plstr = PyObject_Str(plval);<br />
! if (!plstr)<br /> ! PLy_elog(ERROR, "could not compute string representation of
Pythonobject in PL/Python function \"%s\" while modifying trigger row",<br /> !
proc->proname);<br/> ! src = PyString_AsString(plstr);<br /> ! <br /> !
modvalues[i]=<br /> !
InputFunctionCall(&proc->result.out.r.atts[atti].typfunc,<br/> !
src,<br/> !
proc->result.out.r.atts[atti].typioparam,<br/> !
tupdesc->attrs[atti]->atttypmod);<br/> ! modnulls[i] = '
';<br/> ! <br /> ! Py_DECREF(plstr);<br /> ! plstr = NULL;<br /> ! }<br />
! else<br /> ! {<br /> ! modvalues[i] =<br /> !
InputFunctionCall(&proc->result.out.r.atts[atti].typfunc,<br/> !
NULL,<br/> !
proc->result.out.r.atts[atti].typioparam,<br/> !
tupdesc->attrs[atti]->atttypmod);<br/> ! modnulls[i] =
'n';<br/> }<br /> <br /> Py_DECREF(plval);<br /> --- 556,565 ----<br />
modvalues[i]= (Datum) 0;<br /> modnulls[i] = 'n';<br /> }<br /> !
else<br /> {<br /> ! PLyObToDatum *att =
&proc->result.out.r.atts[atti];<br/> ! modvalues[i] = (att->func) (proc, att, plval,
&modnulls[i]);<br/> }<br /> <br /> Py_DECREF(plval);<br /> ***************<br /> ***
784,791****<br /> Datum rv;<br /> PyObject *volatile plargs = NULL;<br /> PyObject
*volatileplrv = NULL;<br /> - PyObject *volatile plrv_so = NULL;<br /> - char *plrv_sc;<br /> <br
/> PG_TRY();<br /> {<br /> --- 785,790 ----<br /> ***************<br /> *** 862,868 ****<br /> <br />
Py_XDECREF(plargs);<br/> Py_XDECREF(plrv);<br /> -
Py_XDECREF(plrv_so);<br/> <br /> PLy_function_delete_args(proc);<br /> <br /> ---
861,866----<br /> ***************<br /> *** 876,922 ****<br /> }<br /> }<br /> <br /> !
/*<br/> ! * If the function is declared to return void, the Python return value<br /> ! *
mustbe None. For void-returning functions, we also treat a None<br /> ! * return value as a special "void
datum"rather than NULL (as is the<br /> ! * case for non-void-returning functions).<br /> ! */<br />
! if (proc->result.out.d.typoid == VOIDOID)<br /> ! {<br /> ! if (plrv != Py_None)<br />
! ereport(ERROR,<br /> ! (errcode(ERRCODE_DATATYPE_MISMATCH),<br /> !
errmsg("PL/Pythonfunction with return type \"void\" did not return None")));<br /> ! <br /> !
fcinfo->isnull= false;<br /> ! rv = (Datum) 0;<br /> ! }<br /> ! else if
(plrv== Py_None)<br /> ! {<br /> ! fcinfo->isnull = true;<br /> ! if
(proc->result.is_rowtype< 1)<br /> ! rv =
InputFunctionCall(&proc->result.out.d.typfunc,<br/> ! NULL,<br /> !
proc->result.out.d.typioparam,<br/> !
-1);<br/> ! else<br /> ! /* Tuple as None */<br /> !
rv= (Datum) NULL;<br /> ! }<br /> ! else if (proc->result.is_rowtype >= 1)<br />
{<br/> HeapTuple tuple = NULL;<br /> <br /> ! if (PySequence_Check(plrv))<br
/> /* composite type as sequence (tuple, list etc) */<br /> ! tuple =
PLySequence_ToTuple(&proc->result,plrv);<br /> else if (PyMapping_Check(plrv))<br />
/*composite type as mapping (currently only dict) */<br /> ! tuple =
PLyMapping_ToTuple(&proc->result,plrv);<br /> else<br /> /* returned as smth,
mustprovide method __getattr__(name) */<br /> ! tuple = PLyObject_ToTuple(&proc->result,
plrv);<br/> <br /> if (tuple != NULL)<br /> {<br /> --- 874,895 ----<br />
}<br/> }<br /> <br /> ! /* Convert python return value into postgres datatypes */<br
/>! if (proc->result.is_rowtype >= 1)<br /> {<br /> HeapTuple tuple = NULL;<br
/> <br /> ! if (plrv == Py_None)<br /> ! tuple = NULL;<br /> ! else if
(PySequence_Check(plrv))<br/> /* composite type as sequence (tuple, list etc) */<br /> !
tuple= PLySequence_ToTuple(proc, plrv);<br /> else if (PyMapping_Check(plrv))<br />
/*composite type as mapping (currently only dict) */<br /> ! tuple =
PLyMapping_ToTuple(proc,plrv);<br /> else<br /> /* returned as smth, must provide
method__getattr__(name) */<br /> ! tuple = PLyObject_ToTuple(proc, plrv);<br /> <br />
if(tuple != NULL)<br /> {<br /> ***************<br /> *** 931,952 ****<br /> }<br
/> else<br /> {<br /> ! fcinfo->isnull = false;<br /> ! plrv_so =
PyObject_Str(plrv);<br/> ! if (!plrv_so)<br /> ! PLy_elog(ERROR, "could not create string
representationof Python object in PL/Python function \"%s\" while creating return value", proc->proname);<br /> !
plrv_sc= PyString_AsString(plrv_so);<br /> ! rv =
InputFunctionCall(&proc->result.out.d.typfunc,<br/> ! plrv_sc,<br /> !
proc->result.out.d.typioparam,<br/> ! -1);<br
/> }<br /> }<br /> PG_CATCH();<br /> {<br /> Py_XDECREF(plargs);<br />
Py_XDECREF(plrv);<br/> - Py_XDECREF(plrv_so);<br /> <br /> PG_RE_THROW();<br />
}<br/> --- 904,919 ----<br /> }<br /> else<br /> {<br /> ! rv =
(proc->result.out.d.func)(proc,<br /> ! &proc->result.out.d, <br
/>! plrv,<br /> !
&fcinfo->isnull);<br/> }<br /> }<br />
PG_CATCH();<br/> {<br /> Py_XDECREF(plargs);<br /> Py_XDECREF(plrv);<br /> <br />
PG_RE_THROW();<br/> }<br /> ***************<br /> *** 954,960 ****<br /> <br />
Py_XDECREF(plargs);<br/> Py_DECREF(plrv);<br /> - Py_XDECREF(plrv_so);<br /> <br /> return
rv;<br/> }<br /> --- 921,926 ----<br /> ***************<br /> *** 1037,1048 ****<br /> arg =
NULL;<br/> else<br /> {<br /> ! char *ct;<br /> ! <br />
! ct = OutputFunctionCall(&(proc->args[i].in.d.typfunc),<br /> !
fcinfo->arg[i]);<br/> ! arg =
(proc->args[i].in.d.func)(ct);<br /> ! pfree(ct);<br /> }<br />
}<br/> <br /> --- 1003,1010 ----<br /> arg = NULL;<br />
else<br/> {<br /> ! arg = (proc->args[i].in.d.func)
(&(proc->args[i].in.d),<br/> ! fcinfo->arg[i]);<br />
}<br/> }<br /> <br /> ***************<br /> *** 1593,1598 ****<br /> --- 1555,1589
----<br/> arg->typoid = HeapTupleGetOid(typeTup);<br /> arg->typioparam = getTypeIOParam(typeTup);<br
/> arg->typbyval = typeStruct->typbyval;<br /> + <br /> + /* Determine which kind of Python object we
willconvert to */<br /> + switch (arg->typoid)<br /> + {<br /> + case VOIDOID:<br /> +
arg->func= PLyObject_ToVoid;<br /> + break;<br /> + case BOOLOID:<br /> +
arg->func= PLyObject_ToBool;<br /> + break;<br /> + case BYTEAOID:<br /> +
arg->func= PLyObject_ToBytea;<br /> + break;<br /> + case BPCHAROID:<br /> +
caseVARCHAROID:<br /> + case TEXTOID:<br /> + arg->func = PLyObject_ToText;<br /> +
break;<br/> + <br /> + case FLOAT4OID:<br /> + case FLOAT8OID:<br /> + case
NUMERICOID:<br/> + case INT2OID:<br /> + case INT4OID:<br /> + case INT8OID:<br /> +
default:<br/> + arg->func = PLyObject_ToDatum;<br /> + break;<br /> + }<br />
}<br/> <br /> static void<br /> ***************<br /> *** 1619,1644 ****<br /> switch (typeOid)<br />
{<br/> case BOOLOID:<br /> ! arg->func = PLyBool_FromString;<br />
break;<br/> case FLOAT4OID:<br /> case FLOAT8OID:<br /> case NUMERICOID:<br
/>! arg->func = PLyFloat_FromString;<br /> break;<br /> case INT2OID:<br />
caseINT4OID:<br /> ! arg->func = PLyInt_FromString;<br /> break;<br />
caseINT8OID:<br /> ! arg->func = PLyLong_FromString;<br /> break;<br />
default:<br/> ! arg->func = PLyString_FromString;<br /> break;<br /> }<br
/> }<br /> <br /> static void<br /> PLy_typeinfo_init(PLyTypeInfo * arg)<br /> {<br /> --- 1610,1648 ----<br
/> switch (typeOid)<br /> {<br /> case BOOLOID:<br /> ! arg->func =
PLyBool_FromBool;<br/> break;<br /> case FLOAT4OID:<br /> + arg->func =
PLyFloat_FromFloat4;<br/> + break;<br /> case FLOAT8OID:<br /> + arg->func =
PLyFloat_FromFloat8;<br/> + break;<br /> case NUMERICOID:<br /> ! arg->func =
PLyFloat_FromNumeric;<br/> break;<br /> case INT2OID:<br /> + arg->func =
PLyInt_FromInt16;<br/> + break;<br /> case INT4OID:<br /> ! arg->func =
PLyInt_FromInt32;<br/> break;<br /> case INT8OID:<br /> ! arg->func =
PLyLong_FromInt64;<br/> ! break;<br /> ! case BPCHAROID:<br /> ! case VARCHAROID:<br /> !
caseTEXTOID:<br /> ! case BYTEAOID:<br /> ! arg->func = PLyString_FromText;<br />
break;<br/> default:<br /> ! arg->func = PLyString_FromDatum;<br />
break;<br/> }<br /> }<br /> <br /> + <br /> static void<br /> PLy_typeinfo_init(PLyTypeInfo
*arg)<br /> {<br /> ***************<br /> *** 1660,1716 ****<br /> }<br /> }<br /> <br /> - /* assumes that
abool is always returned as a 't' or 'f' */<br /> static PyObject *<br /> ! PLyBool_FromString(const char *src)<br />
{<br/> /*<br /> * We would like to use Py_RETURN_TRUE and Py_RETURN_FALSE here for<br /> *
generatingSQL from trigger functions, but those are only supported in<br /> * Python >= 2.3, and we support
olderversions.<br /> * <a
href="http://docs.python.org/api/boolObjects.html">http://docs.python.org/api/boolObjects.html</a><br/> */<br />
! if (src[0] == 't')<br /> return PyBool_FromLong(1);<br /> ! return PyBool_FromLong(0);<br /> }<br
/> <br /> static PyObject *<br /> ! PLyFloat_FromString(const char *src)<br /> {<br /> ! double v;<br
/>! char *eptr;<br /> <br /> ! errno = 0;<br /> ! v = strtod(src, &eptr);<br /> ! if (*eptr
!='\0' || errno)<br /> ! return NULL;<br /> ! return PyFloat_FromDouble(v);<br /> }<br /> <br />
staticPyObject *<br /> ! PLyInt_FromString(const char *src)<br /> {<br /> ! long v;<br /> ! char
*eptr;<br/> <br /> ! errno = 0;<br /> ! v = strtol(src, &eptr, 0);<br /> ! if (*eptr != '\0' ||
errno)<br/> ! return NULL;<br /> ! return PyInt_FromLong(v);<br /> }<br /> <br /> static PyObject
*<br/> ! PLyLong_FromString(const char *src)<br /> {<br /> ! return PyLong_FromString((char *) src, NULL, 0);<br
/> }<br /> <br /> static PyObject *<br /> ! PLyString_FromString(const char *src)<br /> {<br /> ! return
PyString_FromString(src);<br/> }<br /> <br /> static PyObject *<br /> --- 1664,1758 ----<br /> }<br />
}<br/> <br /> static PyObject *<br /> ! PLyBool_FromBool(PLyDatumToOb *arg, Datum d)<br /> {<br /> + bool x
=DatumGetBool(d);<br /> + arg = 0; /* unused */<br /> + <br /> /*<br /> * We would like to use
Py_RETURN_TRUEand Py_RETURN_FALSE here for<br /> * generating SQL from trigger functions, but those are only
supportedin<br /> * Python >= 2.3, and we support older versions.<br /> * <a
href="http://docs.python.org/api/boolObjects.html">http://docs.python.org/api/boolObjects.html</a><br/> */<br />
! if (x)<br /> return PyBool_FromLong(1);<br /> ! else<br /> ! return PyBool_FromLong(0);<br
/> }<br /> <br /> static PyObject *<br /> ! PLyFloat_FromFloat4(PLyDatumToOb *arg, Datum d)<br /> {<br /> !
arg= 0; /* unused */<br /> ! return PyFloat_FromDouble(DatumGetFloat4(d));<br /> ! }<br /> <br /> ! static
PyObject*<br /> ! PLyFloat_FromFloat8(PLyDatumToOb *arg, Datum d)<br /> ! {<br /> ! arg = 0; /* unused */<br /> !
returnPyFloat_FromDouble(DatumGetFloat8(d));<br /> }<br /> <br /> static PyObject *<br /> !
PLyFloat_FromNumeric(PLyDatumToOb*arg, Datum d)<br /> {<br /> ! /* <br /> ! * Numeric is cast to a PyFloat:
<br/> ! * This results in a loss of precision<br /> ! * Would it be better to cast to PyString? <br /> !
*/<br/> ! Datum f = DirectFunctionCall1(numeric_float8, d);<br /> ! double x = DatumGetFloat8(f);<br /> !
arg= 0; /* unused */<br /> ! return PyFloat_FromDouble(x);<br /> ! }<br /> <br /> ! static PyObject *<br />
!PLyInt_FromInt16(PLyDatumToOb *arg, Datum d)<br /> ! {<br /> ! arg = 0; /* unused */<br /> ! return
PyInt_FromLong(DatumGetInt16(d));<br/> }<br /> <br /> static PyObject *<br /> ! PLyInt_FromInt32(PLyDatumToOb
*arg,Datum d)<br /> {<br /> ! arg = 0; /* unused */<br /> ! return PyInt_FromLong(DatumGetInt32(d));<br />
}<br/> <br /> static PyObject *<br /> ! PLyLong_FromInt64(PLyDatumToOb *arg, Datum d)<br /> {<br /> ! arg =
0; /* unused */<br /> ! <br /> ! /* on 32 bit platforms "long" may be too small */<br /> ! if (sizeof(int64)
>sizeof(long))<br /> ! return PyLong_FromLongLong(DatumGetInt64(d));<br /> ! else<br /> ! return
PyLong_FromLong(DatumGetInt64(d));<br/> ! }<br /> ! <br /> ! static PyObject *<br /> ! PLyString_FromText(PLyDatumToOb
*arg,Datum d)<br /> ! {<br /> ! text *txt = DatumGetTextP(d);<br /> ! char *str = VARDATA(txt);<br /> !
size_t size = VARSIZE(txt) - VARHDRSZ;<br /> ! <br /> ! return PyString_FromStringAndSize(str, size);<br /> !
}<br/> ! <br /> ! static PyObject *<br /> ! PLyString_FromDatum(PLyDatumToOb *arg, Datum d)<br /> ! {<br /> ! char
*x= OutputFunctionCall(&arg->typfunc, d);<br /> ! PyObject *r = PyString_FromString(x);<br /> !
pfree(x);<br/> ! return r;<br /> }<br /> <br /> static PyObject *<br /> ***************<br /> ***
1730,1737****<br /> {<br /> for (i = 0; i < info->in.r.natts; i++)<br /> {<br /> !
char *key,<br /> ! *vsrc;<br /> Datum vattr;<br />
bool is_null;<br /> PyObject *value;<br /> --- 1772,1778 ----<br /> {<br />
for(i = 0; i < info->in.r.natts; i++)<br /> {<br /> ! char *key;<br />
Datum vattr;<br /> bool is_null;<br /> PyObject *value;<br />
***************<br/> *** 1746,1759 ****<br /> PyDict_SetItemString(dict, key, Py_None);<br />
else<br/> {<br /> ! vsrc =
OutputFunctionCall(&info->in.r.atts[i].typfunc,<br/> ! vattr);<br /> !
<br/> ! /*<br /> ! * no exceptions allowed<br /> ! */<br /> !
value= info->in.r.atts[i].func(vsrc);<br /> ! pfree(vsrc);<br />
PyDict_SetItemString(dict,key, value);<br /> Py_DECREF(value);<br />
}<br/> --- 1787,1793 ----<br /> PyDict_SetItemString(dict, key, Py_None);<br />
else<br/> {<br /> ! value = (info->in.r.atts[i].func)
(&info->in.r.atts[i],vattr);<br /> PyDict_SetItemString(dict, key, value);<br />
Py_DECREF(value);<br/> }<br /> ***************<br /> *** 1769,1777 ****<br />
returndict;<br /> }<br /> <br /> <br /> static HeapTuple<br /> ! PLyMapping_ToTuple(PLyTypeInfo * info,
PyObject* mapping)<br /> {<br /> TupleDesc desc;<br /> HeapTuple tuple;<br /> --- 1803,2017 ----<br
/> return dict;<br /> }<br /> <br /> + static Datum <br /> + PLyObject_ToVoid(PLyProcedure *proc, <br /> +
PLyObToDatum*arg, <br /> + PyObject *plrv, <br /> + bool *isnull)<br
/>+ {<br /> + /* <br /> + * If the function is declared to return void, the Python return value must<br /> +
*be None. For void-returning functions, we also treat a None return value<br /> + * as a special "void
datum"rather than NULL (as is the case for the <br /> + * non-void-returning functions).<br /> + */<br /> +
if(plrv != Py_None)<br /> + ereport(ERROR,<br /> + (errcode(ERRCODE_DATATYPE_MISMATCH),<br
/>+ errmsg("PL/Python function with return type \"void\" did not "<br /> +
"returnNone")));<br /> + <br /> + *isnull = false;<br /> + return (Datum) 0;<br /> +
}<br/> + <br /> + static Datum <br /> + PLyObject_ToBool(PLyProcedure *proc, <br /> + PLyObToDatum
*arg,<br /> + PyObject *plrv, <br /> + bool *isnull)<br /> + {<br /> + bool rv;
<br/> + <br /> + if (plrv == Py_None)<br /> + {<br /> + *isnull = true;<br /> + return (Datum)
0;<br/> + }<br /> + <br /> + rv = PyObject_IsTrue(plrv);<br /> + *isnull = false;<br /> + return
BoolGetDatum(rv);<br/> + }<br /> + <br /> + <br /> + static Datum <br /> + PLyObject_ToBytea(PLyProcedure *proc, <br />
+ PLyObToDatum *arg, <br /> + PyObject *plrv, <br /> + bool
*isnull)<br/> + {<br /> + PyObject *volatile plrv_so = NULL;<br /> + Datum rv;<br /> + <br /> + if
(plrv== Py_None)<br /> + {<br /> + *isnull = true;<br /> + return (Datum) 0;<br /> + }<br /> +
<br/> + plrv_so = PyObject_Str(plrv);<br /> + if (!plrv_so)<br /> + {<br /> + ereport(ERROR,<br />
+ (errcode(ERRCODE_DATATYPE_MISMATCH),<br /> + errmsg("could not create string
representationof Python "<br /> + "object in PL/Python function \"%s\" while creating "<br /> +
"returnvalue", proc->proname)));<br /> + }<br /> + <br /> + PG_TRY();<br /> +
{<br/> + char *plrv_sc = PyString_AsString(plrv_so);<br /> + size_t len =
PyString_Size(plrv_so);<br/> + size_t size = len + VARHDRSZ;<br /> + bytea *result = (bytea*)
palloc(size);<br/> + <br /> + SET_VARSIZE(result, size);<br /> + memcpy(VARDATA(result), plrv_sc,
len);<br/> + rv = PointerGetDatum(result);<br /> + }<br /> + PG_CATCH();<br /> + {<br /> +
Py_XDECREF(plrv_so);<br/> + PG_RE_THROW();<br /> + }<br /> + PG_END_TRY();<br /> + <br /> +
Py_XDECREF(plrv_so);<br/> + <br /> + *isnull = false;<br /> + return rv;<br /> + }<br /> + <br /> + static
Datum<br /> + PLyObject_ToText(PLyProcedure *proc, <br /> + PLyObToDatum *arg, <br /> +
PyObject*plrv, <br /> + bool *isnull)<br /> + {<br /> + PyObject *volatile
plrv_so= NULL;<br /> + Datum rv;<br /> + <br /> + if (plrv == Py_None)<br /> + {<br /> +
*isnull= true;<br /> + return (Datum) 0;<br /> + }<br /> + <br /> + plrv_so =
PyObject_Str(plrv);<br/> + if (!plrv_so)<br /> + {<br /> + ereport(ERROR,<br /> +
(errcode(ERRCODE_DATATYPE_MISMATCH),<br/> + errmsg("could not create string
representationof Python "<br /> + "object in PL/Python function \"%s\" while creating "<br /> +
"returnvalue", proc->proname)));<br /> + }<br /> + <br /> + PG_TRY();<br /> +
{<br/> + char *plrv_sc = PyString_AsString(plrv_so);<br /> + size_t len =
PyString_Size(plrv_so);<br/> + size_t size = len + VARHDRSZ;<br /> + text *result;<br /> + <br /> +
if(strlen(plrv_sc) != (size_t) len)<br /> + {<br /> + ereport(ERROR,<br /> +
(errcode(ERRCODE_DATATYPE_MISMATCH),<br/> + errmsg("PL/Python function \"%s\"
couldnot convert "<br /> + "Python object into text: expected string without "<br /> +
"nullbytes", proc->proname)));<br /> + }<br /> + <br /> + result =
(bytea*)palloc(size);<br /> + SET_VARSIZE(result, size);<br /> + memcpy(VARDATA(result), plrv_sc,
len);<br/> + rv = PointerGetDatum(result);<br /> + }<br /> + PG_CATCH();<br /> + {<br /> +
Py_XDECREF(plrv_so);<br/> + PG_RE_THROW();<br /> + }<br /> + PG_END_TRY();<br /> + <br /> +
Py_XDECREF(plrv_so);<br/> + <br /> + *isnull = false;<br /> + return rv;<br /> + }<br /> + <br /> + /* <br
/>+ * Generic conversion function:<br /> + * - Cast PyObject to cstring and cstring into postgres type.<br /> +
*/<br/> + static Datum <br /> + PLyObject_ToDatum(PLyProcedure *proc, <br /> + PLyObToDatum *arg,
<br/> + PyObject *plrv, <br /> + bool *isnull)<br /> + {<br /> + PyObject
*volatileplrv_so = NULL;<br /> + Datum rv;<br /> + <br /> + if (plrv == Py_None)<br /> + {<br /> +
*isnull= true;<br /> + return (Datum) 0;<br /> + }<br /> + <br /> + plrv_so =
PyObject_Str(plrv);<br/> + if (!plrv_so)<br /> + {<br /> + ereport(ERROR,<br /> +
(errcode(ERRCODE_DATATYPE_MISMATCH),<br/> + errmsg("could not create string
representationof Python "<br /> + "object in PL/Python function \"%s\" while creating "<br /> +
"returnvalue", proc->proname)));<br /> + }<br /> + <br /> + PG_TRY();<br /> +
{<br/> + char *plrv_sc = PyString_AsString(plrv_so);<br /> + size_t len =
PyString_Size(plrv_so); <br /> + <br /> + if (strlen(plrv_sc) != (size_t) len)<br /> + {<br /> +
ereport(ERROR,<br/> + (errcode(ERRCODE_DATATYPE_MISMATCH),<br /> +
errmsg("PL/Pythonfunction \"%s\" could not convert "<br /> + "Python
objectinto cstring: expected string without "<br /> + "null bytes", proc->proname)));<br
/>+ }<br /> + rv = InputFunctionCall(&arg->typfunc, plrv_sc, arg->typioparam, -1);<br /> +
}<br/> + PG_CATCH();<br /> + {<br /> + Py_XDECREF(plrv_so);<br /> + PG_RE_THROW();<br /> +
}<br/> + PG_END_TRY();<br /> + <br /> + Py_XDECREF(plrv_so);<br /> + <br /> + *isnull = false;<br /> +
returnrv;<br /> + }<br /> <br /> static HeapTuple<br /> ! PLyMapping_ToTuple(PLyProcedure *proc, PyObject
*mapping)<br/> {<br /> TupleDesc desc;<br /> HeapTuple tuple;<br /> ***************<br /> ***
1781,1840****<br /> <br /> Assert(PyMapping_Check(mapping));<br /> <br /> ! desc =
lookup_rowtype_tupdesc(info->out.d.typoid,-1);<br /> ! if (info->is_rowtype == 2)<br /> !
PLy_output_tuple_funcs(info,desc);<br /> ! Assert(info->is_rowtype == 1);<br /> <br /> /* Build
tuple*/<br /> values = palloc(sizeof(Datum) * desc->natts);<br /> nulls = palloc(sizeof(bool) *
desc->natts);<br/> for (i = 0; i < desc->natts; ++i)<br /> {<br /> ! char *key;<br
/>! PyObject *volatile value,<br /> ! *volatile so;<br /> <br /> key =
NameStr(desc->attrs[i]->attname);<br/> ! value = so = NULL;<br /> PG_TRY();<br />
{<br/> value = PyMapping_GetItemString(mapping, key);<br /> ! if (value ==
Py_None)<br/> {<br /> - values[i] = (Datum) NULL;<br /> - nulls[i] =
true;<br/> - }<br /> - else if (value)<br /> - {<br /> - char
*valuestr;<br/> - <br /> - so = PyObject_Str(value);<br /> - if (so == NULL)<br
/>- PLy_elog(ERROR, "could not compute string representation of Python object");<br /> -
valuestr= PyString_AsString(so);<br /> - <br /> - values[i] =
InputFunctionCall(&info->out.r.atts[i].typfunc<br/> - ,valuestr<br
/>- ,info->out.r.atts[i].typioparam<br /> -
,-1);<br/> - Py_DECREF(so);<br /> - so =
NULL;<br/> - nulls[i] = false;<br /> - }<br /> - else<br />
ereport(ERROR,<br/> (errcode(ERRCODE_UNDEFINED_COLUMN),<br />
errmsg("key\"%s\" not found in mapping", key),<br /> errhint("To
returnnull in a column, "<br /> ! "add the value None to the mapping with the key named after the
column.")));<br/> <br /> Py_XDECREF(value);<br /> value = NULL;<br /> }<br />
PG_CATCH();<br/> {<br /> - Py_XDECREF(so);<br /> Py_XDECREF(value);<br />
PG_RE_THROW();<br/> }<br /> --- 2021,2062 ----<br /> <br />
Assert(PyMapping_Check(mapping));<br/> <br /> ! desc = lookup_rowtype_tupdesc(proc->result.out.d.typoid,
-1);<br/> ! if (proc->result.is_rowtype == 2)<br /> ! PLy_output_tuple_funcs(&proc->result,
desc);<br/> ! Assert(proc->result.is_rowtype == 1);<br /> <br /> /* Build tuple */<br /> values =
palloc(sizeof(Datum)* desc->natts);<br /> nulls = palloc(sizeof(bool) * desc->natts);<br /> for (i =
0;i < desc->natts; ++i)<br /> {<br /> ! char *key;<br /> ! PLyObToDatum *att;<br />
! PyObject *volatile value;<br /> <br /> + att = &proc->result.out.r.atts[i];<br />
key= NameStr(desc->attrs[i]->attname);<br /> ! value = NULL;<br /> PG_TRY();<br />
{<br/> value = PyMapping_GetItemString(mapping, key);<br /> ! if (!value)<br />
{<br/> ereport(ERROR,<br />
(errcode(ERRCODE_UNDEFINED_COLUMN),<br/> errmsg("key \"%s\" not
foundin mapping", key),<br /> errhint("To return null in a column, "<br /> !
"addthe value None to the mapping with the "<br /> !
"keynamed after the column.")));<br /> ! }<br /> ! values[i] =
(att->func)(proc, att, value, &nulls[i]);<br /> <br /> Py_XDECREF(value);<br />
value= NULL;<br /> }<br /> PG_CATCH();<br /> {<br />
Py_XDECREF(value);<br/> PG_RE_THROW();<br /> }<br /> ***************<br /> ***
1851,1857****<br /> <br /> <br /> static HeapTuple<br /> ! PLySequence_ToTuple(PLyTypeInfo * info, PyObject *
sequence)<br/> {<br /> TupleDesc desc;<br /> HeapTuple tuple;<br /> --- 2073,2079 ----<br /> <br
/> <br /> static HeapTuple<br /> ! PLySequence_ToTuple(PLyProcedure *proc, PyObject *sequence)<br /> {<br />
TupleDesc desc;<br /> HeapTuple tuple;<br /> ***************<br /> *** 1866,1922 ****<br /> *
canignore exceeding items or assume missing ones as null but to avoid<br /> * plpython developer's errors we are
stricthere<br /> */<br /> ! desc = lookup_rowtype_tupdesc(info->out.d.typoid, -1);<br /> if
(PySequence_Length(sequence)!= desc->natts)<br /> ereport(ERROR,<br />
(errcode(ERRCODE_DATATYPE_MISMATCH),<br/> errmsg("length of returned sequence did not match
numberof columns in row")));<br /> <br /> ! if (info->is_rowtype == 2)<br /> !
PLy_output_tuple_funcs(info,desc);<br /> ! Assert(info->is_rowtype == 1);<br /> <br /> /* Build
tuple*/<br /> values = palloc(sizeof(Datum) * desc->natts);<br /> nulls = palloc(sizeof(bool) *
desc->natts);<br/> for (i = 0; i < desc->natts; ++i)<br /> {<br /> ! PyObject *volatile
value,<br/> ! *volatile so;<br /> <br /> ! value = so = NULL;<br /> PG_TRY();<br
/> {<br /> value = PySequence_GetItem(sequence, i);<br /> Assert(value);<br /> !
if(value == Py_None)<br /> ! {<br /> ! values[i] = (Datum) NULL;<br /> !
nulls[i]= true;<br /> ! }<br /> ! else if (value)<br /> ! {<br /> !
char *valuestr;<br /> ! <br /> ! so = PyObject_Str(value);<br /> !
if(so == NULL)<br /> ! PLy_elog(ERROR, "could not compute string representation of
Pythonobject");<br /> ! valuestr = PyString_AsString(so);<br /> ! values[i] =
InputFunctionCall(&info->out.r.atts[i].typfunc<br/> ! ,valuestr<br
/>! ,info->out.r.atts[i].typioparam<br /> !
,-1);<br/> ! Py_DECREF(so);<br /> ! so =
NULL;<br/> ! nulls[i] = false;<br /> ! }<br /> <br /> Py_XDECREF(value);<br
/> value = NULL;<br /> }<br /> PG_CATCH();<br /> {<br /> -
Py_XDECREF(so);<br/> Py_XDECREF(value);<br /> PG_RE_THROW();<br />
}<br/> --- 2088,2124 ----<br /> * can ignore exceeding items or assume missing ones as null but to
avoid<br/> * plpython developer's errors we are strict here<br /> */<br /> ! desc =
lookup_rowtype_tupdesc(proc->result.out.d.typoid,-1);<br /> if (PySequence_Length(sequence) !=
desc->natts)<br/> ereport(ERROR,<br /> (errcode(ERRCODE_DATATYPE_MISMATCH),<br />
errmsg("lengthof returned sequence did not match number of columns in row")));<br /> <br /> ! if
(proc->result.is_rowtype== 2)<br /> ! PLy_output_tuple_funcs(&proc->result, desc);<br /> !
Assert(proc->result.is_rowtype== 1);<br /> <br /> /* Build tuple */<br /> values =
palloc(sizeof(Datum)* desc->natts);<br /> nulls = palloc(sizeof(bool) * desc->natts);<br /> for (i =
0;i < desc->natts; ++i)<br /> {<br /> ! PLyObToDatum *att;<br /> ! PyObject *volatile
value;<br/> <br /> ! att = &proc->result.out.r.atts[i];<br /> ! value = NULL;<br />
PG_TRY();<br/> {<br /> value = PySequence_GetItem(sequence, i);<br />
Assert(value);<br/> ! values[i] = (att->func) (proc, att, value, &nulls[i]);<br />
<br/> Py_XDECREF(value);<br /> value = NULL;<br /> }<br />
PG_CATCH();<br/> {<br /> Py_XDECREF(value);<br /> PG_RE_THROW();<br />
}<br/> ***************<br /> *** 1933,1939 ****<br /> <br /> <br /> static HeapTuple<br /> !
PLyObject_ToTuple(PLyTypeInfo* info, PyObject * object)<br /> {<br /> TupleDesc desc;<br /> HeapTuple
tuple;<br/> --- 2135,2141 ----<br /> <br /> <br /> static HeapTuple<br /> ! PLyObject_ToTuple(PLyProcedure
*proc,PyObject *object)<br /> {<br /> TupleDesc desc;<br /> HeapTuple tuple;<br />
***************<br/> *** 1941,1962 ****<br /> bool *nulls;<br /> volatile int i;<br /> <br /> !
desc= lookup_rowtype_tupdesc(info->out.d.typoid, -1);<br /> ! if (info->is_rowtype == 2)<br /> !
PLy_output_tuple_funcs(info,desc);<br /> ! Assert(info->is_rowtype == 1);<br /> <br /> /* Build
tuple*/<br /> values = palloc(sizeof(Datum) * desc->natts);<br /> nulls = palloc(sizeof(bool) *
desc->natts);<br/> for (i = 0; i < desc->natts; ++i)<br /> {<br /> ! char *key;<br
/>! PyObject *volatile value,<br /> ! *volatile so;<br /> <br /> key =
NameStr(desc->attrs[i]->attname);<br/> ! value = so = NULL;<br /> PG_TRY();<br />
{<br/> value = PyObject_GetAttrString(object, key);<br /> --- 2143,2165 ----<br /> bool
*nulls;<br/> volatile int i;<br /> <br /> ! desc =
lookup_rowtype_tupdesc(proc->result.out.d.typoid,-1);<br /> ! if (proc->result.is_rowtype == 2)<br /> !
PLy_output_tuple_funcs(&proc->result,desc);<br /> ! Assert(proc->result.is_rowtype == 1);<br />
<br/> /* Build tuple */<br /> values = palloc(sizeof(Datum) * desc->natts);<br /> nulls =
palloc(sizeof(bool)* desc->natts);<br /> for (i = 0; i < desc->natts; ++i)<br /> {<br /> !
char *key;<br /> ! PLyObToDatum *att;<br /> ! PyObject *volatile value;<br /> <br
/>+ att = &proc->result.out.r.atts[i];<br /> key = NameStr(desc->attrs[i]->attname);<br
/>! value = NULL;<br /> PG_TRY();<br /> {<br /> value =
PyObject_GetAttrString(object,key);<br /> ***************<br /> *** 1965,2000 ****<br /> values[i] =
(Datum)NULL;<br /> nulls[i] = true;<br /> }<br /> ! else if (value)<br />
{<br/> - char *valuestr;<br /> - <br /> - so =
PyObject_Str(value);<br/> - if (so == NULL)<br /> - PLy_elog(ERROR, "could not
computestring representation of Python object");<br /> - valuestr = PyString_AsString(so);<br /> -
values[i]= InputFunctionCall(&info->out.r.atts[i].typfunc<br /> -
,valuestr<br/> -
,info->out.r.atts[i].typioparam<br/> -
,-1);<br/> - Py_DECREF(so);<br /> - so =
NULL;<br/> - nulls[i] = false;<br /> - }<br /> - else<br />
ereport(ERROR,<br/> (errcode(ERRCODE_UNDEFINED_COLUMN),<br /> !
errmsg("attribute\"%s\" does not exist in Python object", key),<br />
errhint("Toreturn null in a column, "<br /> ! "let the
returnedobject have an attribute named "<br /> ! "after column with value
None.")));<br/> <br /> Py_XDECREF(value);<br /> value = NULL;<br /> }<br />
PG_CATCH();<br/> {<br /> - Py_XDECREF(so);<br /> Py_XDECREF(value);<br />
PG_RE_THROW();<br/> }<br /> --- 2168,2190 ----<br /> values[i] = (Datum)
NULL;<br/> nulls[i] = true;<br /> }<br /> ! else if (!value)<br />
{<br/> ereport(ERROR,<br />
(errcode(ERRCODE_UNDEFINED_COLUMN),<br/> ! errmsg("key \"%s\" not
foundin object", key),<br /> errhint("To return null in a column, "<br /> !
"addthe value None to the mapping with the "<br /> !
"keynamed after the column.")));<br /> ! }<br /> ! else<br />
! values[i] = (att->func) (proc, att, value, &nulls[i]);<br /> <br />
Py_XDECREF(value);<br/> value = NULL;<br /> }<br /> PG_CATCH();<br />
{<br/> Py_XDECREF(value);<br /> PG_RE_THROW();<br /> }<br /> Index:
expected/plpython_function.out<br/> ===================================================================<br /> RCS file:
/projects/cvsroot/pgsql/src/pl/plpython/expected/plpython_function.out,v<br/> retrieving revision 1.12<br /> diff -c
-r1.12plpython_function.out<br /> *** expected/plpython_function.out 3 Apr 2009 16:59:42 -0000 1.12<br /> ---
expected/plpython_function.out 26 May 2009 22:58:52 -0000<br /> ***************<br /> *** 450,452 ****<br /> ---
450,470----<br /> CREATE FUNCTION test_inout_params(first inout text) AS $$<br /> return first + '_inout';<br />
$$LANGUAGE plpythonu;<br /> + CREATE FUNCTION test_type_conversion_bool(x bool) returns bool AS $$ return x $$
languageplpythonu;<br /> + CREATE FUNCTION test_type_conversion_char(x char) returns char AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_int2(x int2) returns int2 AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_int4(x int4) returns int4 AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_int8(x int8) returns int8 AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_float4(x float4) returns float4 AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_float8(x float8) returns float8 AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_numeric(x numeric) returns numeric AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_text(x text) returns text AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_bytea(x bytea) returns bytea AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_marshal() returns bytea AS $$ <br /> + import marshal<br /> + return
marshal.dumps('helloworld')<br /> + $$ language plpythonu;<br /> + CREATE FUNCTION test_type_unmarshal(x bytea) returns
textAS $$<br /> + import marshal<br /> + return marshal.loads(x)<br /> + $$ language plpythonu;<br /> Index:
expected/plpython_test.out<br/> ===================================================================<br /> RCS file:
/projects/cvsroot/pgsql/src/pl/plpython/expected/plpython_test.out,v<br/> retrieving revision 1.8<br /> diff -c -r1.8
plpython_test.out<br/> *** expected/plpython_test.out 3 Apr 2009 16:59:42 -0000 1.8<br /> ---
expected/plpython_test.out 26 May 2009 22:58:52 -0000<br /> ***************<br /> *** 559,561 ****<br /> --- 559,693
----<br/> test_in_inout<br /> (1 row)<br /> <br /> + SELECT * FROM test_type_conversion_bool(true);<br /> +
test_type_conversion_bool<br /> + ---------------------------<br /> + t<br /> + (1 row)<br /> + <br /> + SELECT *
FROMtest_type_conversion_bool(false);<br /> + test_type_conversion_bool <br /> + ---------------------------<br /> +
f<br/> + (1 row)<br /> + <br /> + SELECT * FROM test_type_conversion_bool(null);<br /> + test_type_conversion_bool
<br/> + ---------------------------<br /> + <br /> + (1 row)<br /> + <br /> + SELECT * FROM
test_type_conversion_char('a');<br/> + test_type_conversion_char <br /> + ---------------------------<br /> + a<br />
+(1 row)<br /> + <br /> + SELECT * FROM test_type_conversion_char(null);<br /> + test_type_conversion_char <br /> +
---------------------------<br/> + <br /> + (1 row)<br /> + <br /> + SELECT * FROM
test_type_conversion_int2(100::int2);<br/> + test_type_conversion_int2 <br /> + ---------------------------<br /> +
100<br/> + (1 row)<br /> + <br /> + SELECT * FROM test_type_conversion_int2(null);<br /> +
test_type_conversion_int2<br /> + ---------------------------<br /> + <br /> + (1 row)<br />
+<br /> + SELECT * FROM test_type_conversion_int4(100);<br /> + test_type_conversion_int4 <br /> +
---------------------------<br/> + 100<br /> + (1 row)<br /> + <br /> + SELECT * FROM
test_type_conversion_int4(null);<br/> + test_type_conversion_int4 <br /> + ---------------------------<br /> +
<br/> + (1 row)<br /> + <br /> + SELECT * FROM test_type_conversion_int8(100);<br /> +
test_type_conversion_int8<br /> + ---------------------------<br /> + 100<br /> + (1 row)<br />
+<br /> + SELECT * FROM test_type_conversion_int8(null);<br /> + test_type_conversion_int8 <br /> +
---------------------------<br/> + <br /> + (1 row)<br /> + <br /> + SELECT * FROM
test_type_conversion_float4(100);<br/> + test_type_conversion_float4 <br /> + -----------------------------<br /> +
100<br/> + (1 row)<br /> + <br /> + SELECT * FROM test_type_conversion_float4(null);<br /> +
test_type_conversion_float4<br /> + -----------------------------<br /> + <br /> + (1
row)<br/> + <br /> + SELECT * FROM test_type_conversion_float8(100);<br /> + test_type_conversion_float8 <br /> +
-----------------------------<br/> + 100<br /> + (1 row)<br /> + <br /> + SELECT * FROM
test_type_conversion_float8(null);<br/> + test_type_conversion_float8 <br /> + -----------------------------<br /> +
<br/> + (1 row)<br /> + <br /> + SELECT * FROM test_type_conversion_numeric(100);<br /> +
test_type_conversion_numeric<br /> + ------------------------------<br /> + 100.0<br /> + (1
row)<br/> + <br /> + SELECT * FROM test_type_conversion_numeric(null);<br /> + test_type_conversion_numeric <br /> +
------------------------------<br/> + <br /> + (1 row)<br /> + <br /> + SELECT * FROM
test_type_conversion_text('helloworld');<br /> + test_type_conversion_text <br /> + ---------------------------<br />
+ hello world<br /> + (1 row)<br /> + <br /> + SELECT * FROM test_type_conversion_text(null);<br /> +
test_type_conversion_text<br /> + ---------------------------<br /> + <br /> + (1 row)<br /> + <br /> + SELECT * FROM
test_type_conversion_bytea('helloworld');<br /> + test_type_conversion_bytea <br /> + ----------------------------<br
/>+ hello world<br /> + (1 row)<br /> + <br /> + SELECT * FROM test_type_conversion_bytea(null);<br /> +
test_type_conversion_bytea<br /> + ----------------------------<br /> + <br /> + (1 row)<br /> + <br /> + SELECT
test_type_unmarshal(x)FROM test_type_marshal() x;<br /> + test_type_unmarshal <br /> + ---------------------<br /> +
helloworld<br /> + (1 row)<br /> + <br /> Index: sql/plpython_function.sql<br />
===================================================================<br/> RCS file:
/projects/cvsroot/pgsql/src/pl/plpython/sql/plpython_function.sql,v<br/> retrieving revision 1.12<br /> diff -c -r1.12
plpython_function.sql<br/> *** sql/plpython_function.sql 3 Apr 2009 16:59:43 -0000 1.12<br /> ---
sql/plpython_function.sql 26 May 2009 22:58:52 -0000<br /> ***************<br /> *** 497,499 ****<br /> --- 497,518
----<br/> CREATE FUNCTION test_inout_params(first inout text) AS $$<br /> return first + '_inout';<br /> $$
LANGUAGEplpythonu;<br /> + <br /> + CREATE FUNCTION test_type_conversion_bool(x bool) returns bool AS $$ return x $$
languageplpythonu;<br /> + CREATE FUNCTION test_type_conversion_char(x char) returns char AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_int2(x int2) returns int2 AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_int4(x int4) returns int4 AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_int8(x int8) returns int8 AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_float4(x float4) returns float4 AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_float8(x float8) returns float8 AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_numeric(x numeric) returns numeric AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_text(x text) returns text AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_conversion_bytea(x bytea) returns bytea AS $$ return x $$ language
plpythonu;<br/> + CREATE FUNCTION test_type_marshal() returns bytea AS $$ <br /> + import marshal<br /> + return
marshal.dumps('helloworld')<br /> + $$ language plpythonu;<br /> + CREATE FUNCTION test_type_unmarshal(x bytea) returns
textAS $$<br /> + import marshal<br /> + return marshal.loads(x)<br /> + $$ language plpythonu;<br /> Index:
sql/plpython_test.sql<br/> ===================================================================<br /> RCS file:
/projects/cvsroot/pgsql/src/pl/plpython/sql/plpython_test.sql,v<br/> retrieving revision 1.5<br /> diff -c -r1.5
plpython_test.sql<br/> *** sql/plpython_test.sql 3 Apr 2009 16:59:43 -0000 1.5<br /> --- sql/plpython_test.sql
26May 2009 22:58:52 -0000<br /> ***************<br /> *** 149,151 ****<br /> --- 149,176 ----<br /> -- this
doesn'twork yet :-(<br /> SELECT * FROM test_in_out_params_multi('test_in');<br /> SELECT * FROM
test_inout_params('test_in');<br/> + <br /> + SELECT * FROM test_type_conversion_bool(true);<br /> + SELECT * FROM
test_type_conversion_bool(false);<br/> + SELECT * FROM test_type_conversion_bool(null);<br /> + SELECT * FROM
test_type_conversion_char('a');<br/> + SELECT * FROM test_type_conversion_char(null);<br /> + SELECT * FROM
test_type_conversion_int2(100::int2);<br/> + SELECT * FROM test_type_conversion_int2(null);<br /> + SELECT * FROM
test_type_conversion_int4(100);<br/> + SELECT * FROM test_type_conversion_int4(null);<br /> + SELECT * FROM
test_type_conversion_int8(100);<br/> + SELECT * FROM test_type_conversion_int8(null);<br /> + SELECT * FROM
test_type_conversion_float4(100);<br/> + SELECT * FROM test_type_conversion_float4(null);<br /> + SELECT * FROM
test_type_conversion_float8(100);<br/> + SELECT * FROM test_type_conversion_float8(null);<br /> + SELECT * FROM
test_type_conversion_numeric(100);<br/> + SELECT * FROM test_type_conversion_numeric(null);<br /> + SELECT * FROM
test_type_conversion_text('helloworld');<br /> + SELECT * FROM test_type_conversion_text(null);<br /> + SELECT * FROM
test_type_conversion_bytea('helloworld');<br /> + SELECT * FROM test_type_conversion_bytea(null);<br /> + SELECT
test_type_unmarshal(x)FROM test_type_marshal() x;<br /> + <br /> + <br /></span></font>
On Wednesday 27 May 2009 02:07:33 Caleb Welton wrote: > Patch for plpythonu > > Primary motivation of the attached patch is to support handling bytea > conversion allowing for embedded nulls, which in turn allows for supporting > the marshal module. > > Secondary motivation is slightly improved performance for conversion > routines of basic datatypes that have simple mappings between > postgres/python. > > Primary design is to change the conversion routines from being based on > cstrings to datums, eg: PLyBool_FromString(const char *) => > PLyBool_FromBool(PLyDatumToOb, Datum); Makes sense; please add it to the next commit fest. Are there any compatibility implications, that is, do any of the conversions work differently from before (except when they were broken before, as in the case of bytea)?
<font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:11pt">All data types should map to the same pythonobject types as they did before, so int32->PyInt, int64->PyLong, numeric->PyFloat, etc.<br /><br /> The conversionroutines are slightly different, eg int32 is initialized via PyInt_FromLong() instead of first converting the integerto a string then calling PyInt_FromString, this is a little faster, but shouldn’t result in differences, if anythingthis should be more correct. <br /><br /> Previously numeric->string->PyFloat_FromString, now numeric->double->PyFloat_FromDouble,which makes use of postgres numeric->double routines rather than python string->doubleroutines, and it is conceivable that there are precision variations between the two. My own feeling onthe matter is that PyFloat is the wrong mapping for numeric, but I didn’t want to muddy this patch by changing that. <br/><br /> The main compatibility issue is with Python Strings that contain embedded nulls. Conversion to bytea will nowwork correctly and build the bytea using PyString_FromStringAndSize instead of PyString_FromString. Other datatypes willnow error if the PyString contains embedded nulls, previously they would silently truncate.<br /><br /> Thanks for thecomments,<br /> Caleb<br /><br /><br /> On 5/27/09 3:11 AM, "Peter Eisentraut" <<a href="peter_e@gmx.net">peter_e@gmx.net</a>>wrote:<br /><br /></span></font><blockquote><font face="Calibri, Verdana, Helvetica,Arial"><span style="font-size:11pt">On Wednesday 27 May 2009 02:07:33 Caleb Welton wrote:<br /> > Patch forplpythonu<br /> ><br /> > Primary motivation of the attached patch is to support handling bytea<br /> > conversionallowing for embedded nulls, which in turn allows for supporting<br /> > the marshal module.<br /> ><br />> Secondary motivation is slightly improved performance for conversion<br /> > routines of basic datatypes that havesimple mappings between<br /> > postgres/python.<br /> ><br /> > Primary design is to change the conversionroutines from being based on<br /> > cstrings to datums, eg: PLyBool_FromString(const char *) =><br /> >PLyBool_FromBool(PLyDatumToOb, Datum);<br /><br /> Makes sense; please add it to the next commit fest.<br /><br /> Arethere any compatibility implications, that is, do any of the conversions<br /> work differently from before (except whenthey were broken before, as in the<br /> case of bytea)?<br /><br /></span></font></blockquote>
On Wednesday 27 May 2009 21:53:31 Caleb Welton wrote: > Previously numeric->string->PyFloat_FromString, now > numeric->double->PyFloat_FromDouble, which makes use of postgres > numeric->double routines rather than python string->double routines, and it > is conceivable that there are precision variations between the two. My own > feeling on the matter is that PyFloat is the wrong mapping for numeric, but > I didn't want to muddy this patch by changing that. Yeah, that one had me wondering for a while as well, but as you say it is better to address that separately.
Peter Eisentraut <peter_e@gmx.net> writes:
> On Wednesday 27 May 2009 21:53:31 Caleb Welton wrote:
>> ... My own
>> feeling on the matter is that PyFloat is the wrong mapping for numeric, but
>> I didn't want to muddy this patch by changing that.
> Yeah, that one had me wondering for a while as well, but as you say it is
> better to address that separately.
That was making me itch as well, in my very cursory look at the patch.
Does Python have a saner mapping for it?
regards, tom lane
<font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:11pt">Yes, in Python >= 2.4 there is the Decimaldatatype. <br /><br /> However, unlike the other mappings employed by plpythonu, Decimal requires an import statementto be in scope.<br /><br /> -Caleb<br /><br /> On 5/27/09 2:07 PM, "Tom Lane" <<a href="tgl@sss.pgh.pa.us">tgl@sss.pgh.pa.us</a>>wrote:<br /><br /></span></font><blockquote><font face="Calibri, Verdana,Helvetica, Arial"><span style="font-size:11pt">Peter Eisentraut <<a href="peter_e@gmx.net">peter_e@gmx.net</a>>writes:<br /> > On Wednesday 27 May 2009 21:53:31 Caleb Welton wrote:<br/> >> ... My own<br /> >> feeling on the matter is that PyFloat is the wrong mapping for numeric, but<br/> >> I didn't want to muddy this patch by changing that.<br /><br /> > Yeah, that one had me wondering fora while as well, but as you say it is<br /> > better to address that separately.<br /><br /> That was making me itchas well, in my very cursory look at the patch.<br /> Does Python have a saner mapping for it?<br /><br /> regards,tom lane<br /><br /></span></font></blockquote>
On Wednesday 27 May 2009 02:07:33 Caleb Welton wrote: > Patch for plpythonu This patch doesn't apply; I think it got mangled during email transport. (Tabs changed to spaces, it looks like.) Could you resend the patch as a separate attachment in a way that it doesn't get mangled?
Sorry about that. Here it is again as an attachment.
-Caleb
On 7/16/09 7:16 AM, "Peter Eisentraut" <peter_e@gmx.net> wrote:
-Caleb
On 7/16/09 7:16 AM, "Peter Eisentraut" <peter_e@gmx.net> wrote:
On Wednesday 27 May 2009 02:07:33 Caleb Welton wrote:
> Patch for plpythonu
This patch doesn't apply; I think it got mangled during email transport.
(Tabs changed to spaces, it looks like.) Could you resend the patch as a
separate attachment in a way that it doesn't get mangled?
Attachment
On tis, 2009-05-26 at 16:07 -0700, Caleb Welton wrote: > Patch for plpythonu > > Primary motivation of the attached patch is to support handling bytea > conversion allowing for embedded nulls, which in turn allows for > supporting the marshal module. > > Secondary motivation is slightly improved performance for conversion > routines of basic datatypes that have simple mappings between > postgres/python. > > Primary design is to change the conversion routines from being based > on cstrings to datums, eg: > PLyBool_FromString(const char *) => > PLyBool_FromBool(PLyDatumToOb, Datum); I have reworked this patch a bit and extended the plpython test suite around it. Current copy attached. The remaining problem is that the patch loses domain checking on the return types, because some paths no longer go through the data type's input function. I have marked these places as FIXME, and the regression tests also contain a failing test case for this. What's needed here, I think, is an API that takes a datum plus type information and checks whether the datum is valid within the domain. I haven't found one that is exported, but maybe someone could give a tip.
Attachment
>> Primary motivation of the attached patch is to support handling bytea >> conversion allowing for embedded nulls, which in turn allows for >> supporting the marshal module. >> >> Secondary motivation is slightly improved performance for conversion >> routines of basic datatypes that have simple mappings between >> postgres/python. >> >> Primary design is to change the conversion routines from being based >> on cstrings to datums, eg: >> PLyBool_FromString(const char *) => >> PLyBool_FromBool(PLyDatumToOb, Datum); > > I have reworked this patch a bit and extended the plpython test suite > around it. Current copy attached. > > The remaining problem is that the patch loses domain checking on the > return types, because some paths no longer go through the data type's > input function. I have marked these places as FIXME, and the regression > tests also contain a failing test case for this. > > What's needed here, I think, is an API that takes a datum plus type > information and checks whether the datum is valid within the domain. I > haven't found one that is exported, but maybe someone could give a tip. I see an intersection between the work I'm currently doing on COPY BINARY and this. Basically if you have an INT, you aren't going to make lots of checks. However, for a TEXT, postgres needs to reject it if it has a NULL in it (which doesn't bother Python at all), or if it is has chars which are not valid in the current encoding, etc. Many other types like TIMESTAMP have checks which are absolutely necessary for correctness... > What's needed here, I think, is an API that takes a datum plus type > information and checks whether the datum is valid within the domain. I > haven't found one that is exported, but maybe someone could give a tip. Problems : - If the data you're trying to put in the Datum doesn't fit (example : out of range error, varchar too small, etc), and you want a datum-type-specific function to check your datum and reject it, how are you going to build the datum ? perhaps you can't, since your value doesn't fit. It's a chicken and egg problem : the check function that you expect to reject your invalid datum will not know it's invalid, since you've trimmed it at the edges to make it fit in the required Datum type... - you are going to build a datum that is perhaps valid, and perhaps not, and send this to a function... having known-invalid datums moving around could be not such a good idea... Why not use the copy binary format to communicate between python and pg ? -> you write code to serialize python objects to binary form -> you call the recv function to get a postgres datum -> recv function throws an error if there is any problem -> as a bonus, you release your python object <-> postgres binary code as a separate library so people can use it to output data readable by COPY BINARY and parse COPY BINARY dumps.
On Aug 15, 2009, at 4:44 PM, Peter Eisentraut wrote: > What's needed here, I think, is an API that takes a datum plus type > information and checks whether the datum is valid within the domain. /agree =)
Peter Eisentraut <peter_e@gmx.net> writes:
> The remaining problem is that the patch loses domain checking on the
> return types, because some paths no longer go through the data type's
> input function. I have marked these places as FIXME, and the regression
> tests also contain a failing test case for this.
For the record, I think this entire patch is a bad idea. PLs should not
be so much in bed with the internal representation of datatypes. To
take just one example, this *will* break when/if we change text to carry
some internal locale indicator. There has been absolutely zero evidence
presented to justify that there's a need to break abstraction to gain
performance in this area.
> What's needed here, I think, is an API that takes a datum plus type
> information and checks whether the datum is valid within the domain. I
> haven't found one that is exported, but maybe someone could give a tip.
There isn't one, but maybe you could expose domain_state_setup and
domain_check_input, or some simple wrapper around them.
regards, tom lane
Tom Lane wrote: > > For the record, I think this entire patch is a bad idea. PLs should not > be so much in bed with the internal representation of datatypes. > I thought there was some suggestion in the past that we should move some in that direction. The discussion context was Theo Schlossnagle's complaint about the overhead of passing bytea to and from PLPerl, although that might be ameliorated by the hex gadget. The other major case that would benefit would be passing Array values as the PL's native array type, and Composite values as the PL's associative array type. That would save PL users a lot of highly error-prone coding deconstructing the text representation of such objects. cheers andrew
Andrew Dunstan <andrew@dunslane.net> writes:
> Tom Lane wrote:
>> For the record, I think this entire patch is a bad idea. PLs should not
>> be so much in bed with the internal representation of datatypes.
> I thought there was some suggestion in the past that we should move some
> in that direction.
There's been some discussion about functional improvements like
translating arrays to arrays. I don't know what we'd have to do
to manage that, but possibly some API extensions to the array code
would make it feasible without violating abstractions. The present
patch, however, doesn't appear to have any reason to live other than an
undocumented amount of performance improvement. My feeling about that
is if you're concerned about micro-performance, why are you coding in
python to begin with? It isn't the best choice out there.
regards, tom lane
Peter Eisentraut wrote: > I have reworked this patch a bit and extended the plpython test suite > around it. Current copy attached. I think the errcontext bits should be committed separately to get them out of the way (and to ensure that they get in, regardless of objections to other parts of the patch). -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
On mån, 2009-08-17 at 10:42 -0400, Tom Lane wrote: > For the record, I think this entire patch is a bad idea. PLs should not > be so much in bed with the internal representation of datatypes. To > take just one example, this *will* break when/if we change text to carry > some internal locale indicator. There has been absolutely zero evidence > presented to justify that there's a need to break abstraction to gain > performance in this area. The motivation for this patch has nothing to do with performance. The point is to pass data types into and out of PL/Python sensibly. In particular, passing bytea into and out of PL/Python is currently completely broken, in the sense that what you get in Python is not a byte string that you can process sensibly. We could argue that peeking inside the internal representation of data types might be inappropriate. In which case the solution would be to run the data through the data type output function and have PL/Python parse that back in. That would just be a localized change in the patch, however. (It might be less than ideal for passing float types, perhaps.) We do, however, expose a data types binary format through the binary protocol, so perhaps we should be using the send/recv functions instead of input/output. Which would require hardcoding the bytea binary format, at least. Either of these solutions would probably solve the domains problem, though. Note also that we have historically broken the bytea text format twice as often as the bytea binary format. ;-)
2009/8/18 Peter Eisentraut <peter_e@gmx.net>: > On mån, 2009-08-17 at 10:42 -0400, Tom Lane wrote: >> For the record, I think this entire patch is a bad idea. PLs should not >> be so much in bed with the internal representation of datatypes. To >> take just one example, this *will* break when/if we change text to carry >> some internal locale indicator. There has been absolutely zero evidence >> presented to justify that there's a need to break abstraction to gain >> performance in this area. > I thing, so communication based on text type is bad. Maybe we should to use binary communication based on send and recv function? It's should be better and maybe stable than direct transfer - but maybe little bit slower. > The motivation for this patch has nothing to do with performance. The > point is to pass data types into and out of PL/Python sensibly. In > particular, passing bytea into and out of PL/Python is currently > completely broken, in the sense that what you get in Python is not a > byte string that you can process sensibly. > I thing so bytea should be very well optimized. You can expect, so there be moved bigger block of data. regards Pavel > We could argue that peeking inside the internal representation of data > types might be inappropriate. In which case the solution would be to > run the data through the data type output function and have PL/Python > parse that back in. That would just be a localized change in the patch, > however. (It might be less than ideal for passing float types, > perhaps.) We do, however, expose a data types binary format through the > binary protocol, so perhaps we should be using the send/recv functions > instead of input/output. Which would require hardcoding the bytea > binary format, at least. Either of these solutions would probably solve > the domains problem, though. > > Note also that we have historically broken the bytea text format twice > as often as the bytea binary format. ;-) > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers >
<font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:11pt">As documented in the patch, the primary motivationwas support of BYTEA datatype, which when cast through cstring was truncating python strings with embedded nulls,<br /> performance was only a secondary consideration.<br /><br /> Regards, <br /> Caleb<br /><br /> (Sorry for myslow entry on this thread, I’m on vacation right now.)<br /><br /> On 8/17/09 8:12 AM, "Tom Lane" <<a href="tgl@sss.pgh.pa.us">tgl@sss.pgh.pa.us</a>>wrote:<br /><br /></span></font><blockquote><font face="Calibri, Verdana,Helvetica, Arial"><span style="font-size:11pt">Andrew Dunstan <<a href="andrew@dunslane.net">andrew@dunslane.net</a>>writes:<br /> > Tom Lane wrote:<br /> >> For the record, Ithink this entire patch is a bad idea. PLs should not<br /> >> be so much in bed with the internal representationof datatypes. <br /><br /> > I thought there was some suggestion in the past that we should move some<br/> > in that direction.<br /><br /> There's been some discussion about functional improvements like<br /> translatingarrays to arrays. I don't know what we'd have to do<br /> to manage that, but possibly some API extensions tothe array code<br /> would make it feasible without violating abstractions. The present<br /> patch, however, doesn'tappear to have any reason to live other than an<br /> undocumented amount of performance improvement. My feelingabout that<br /> is if you're concerned about micro-performance, why are you coding in<br /> python to begin with? It isn't the best choice out there.<br /><br /> regards, tom lane<br /><br /></span></font></blockquote>
On Sat, Aug 22, 2009 at 11:45 AM, Caleb Welton<cwelton@greenplum.com> wrote: > As documented in the patch, the primary motivation was support of BYTEA > datatype, which when cast through cstring was truncating python strings with > embedded nulls, > performance was only a secondary consideration. The alternative to attaching to the internal representation would be to marshal and unmarshal the text representation where nuls are escaped as \000. However I dispute this this is "micro-performance" that we're talking about. On any given small datum it may be a small incremental amount of time but it's not incremental time that matters, it's aggregate. If you're processing 1TB of data and you have to marshal and unmarshal all 1TB it doesn't matter that you're doing it in 100 byte chunks. And in any case there are plenty of people throwing around multi-megabyte bytea blobs and having to marshal and unmarshal them every time they go from the database into a PL or back would be a noticeable delay and risk of out-of-memory errors. If we want PLs to not be overly in bed with Postgres data types then the way to do it is to have data types provide abstract methods for accessing their internals. At least for bytea and text that would be fairly straightforward. For numeric I don't see that it would really buy much since it wouldn't really let us completely change representations. -- greg http://mit.edu/~gsstark/resume.pdf
Greg Stark <gsstark@mit.edu> writes:
> On Sat, Aug 22, 2009 at 11:45 AM, Caleb Welton<cwelton@greenplum.com> wrote:
>> As documented in the patch, the primary motivation was support of BYTEA
>> datatype, which when cast through cstring was truncating python strings with
>> embedded nulls,
>> performance was only a secondary consideration.
> The alternative to attaching to the internal representation would be
> to marshal and unmarshal the text representation where nuls are
> escaped as \000.
I don't actually have a problem with depending on the internal
representation of bytea. What I'm unhappy about is that (despite
Caleb's assertions that this is only about bytea) the patch proceeds
to make plpython intimate with the internal representation of a bunch
of *other* datatypes, some of which we have good reason to think may
change in future. If it were only touching bytea I would not have
complained.
regards, tom lane
<font face="Calibri, Verdana, Helvetica, Arial"><span style="font-size:11pt">I didn’t say that it _only_ affects bytea, Isaid that was the _primary motivation_ for it. <br /><br /> Converting from postgres=>python this change affects boolean,float4, float8, numeric, int16, int32, int64, text, and bytea. The code to handle this goes through DatumGetXXXfor the native C type for the datatype, with the exception of the Varlena types (special case) and Numeric whichcalls numeric_float8() to convert the numeric to a native C double precision float. As mentioned in the original postI do not think that this is appropriate for numeric, and I would prefer a better mapping, but this was a pre-existingissue and is not a change in behavior for the patch. Since this is a separate issue I opted not to change itto keep the patch concise. <br /><br /> Converting from python=>postgres this change effects void, bool, bytea, andtext. <br /><br /> The reason for this asymmetry is that there is not a 1:1 mapping of Postgres datatypes to Python datatypesand conciseness of the patch.<br /><br /> All other datatypes (including arrays unfortunately) go through the sametext input functions that they did before.<br /><br /> Of the above I would expect the only type that we would have goodreason to expect to change would be numeric, and this patch _doesn’t_ rely on it’s internal representation: it callsnumeric_float8().<br /><br /> I think it would be good to have mappings for other datatypes, depending on internal representationor not, but thought that was beyond the scope of the patch.<br /><br /> Regards,<br /> Caleb<br /><br />On 8/22/09 7:03 AM, "Tom Lane" <<a href="tgl@sss.pgh.pa.us">tgl@sss.pgh.pa.us</a>> wrote:<br /><br /></span></font><blockquote><fontface="Calibri, Verdana, Helvetica, Arial"><span style="font-size:11pt">Greg Stark <<ahref="gsstark@mit.edu">gsstark@mit.edu</a>> writes:<br /> > On Sat, Aug 22, 2009 at 11:45 AM, Caleb Welton<<ahref="cwelton@greenplum.com">cwelton@greenplum.com</a>> wrote:<br /> >> As documented in the patch,the primary motivation was support of BYTEA<br /> >> datatype, which when cast through cstring was truncatingpython strings with<br /> >> embedded nulls,<br /> >> performance was only a secondary consideration.<br/><br /> > The alternative to attaching to the internal representation would be<br /> > to marshaland unmarshal the text representation where nuls are<br /> > escaped as \000.<br /><br /> I don't actually havea problem with depending on the internal<br /> representation of bytea. What I'm unhappy about is that (despite<br />Caleb's assertions that this is only about bytea) the patch proceeds<br /> to make plpython intimate with the internalrepresentation of a bunch<br /> of *other* datatypes, some of which we have good reason to think may<br /> changein future. If it were only touching bytea I would not have<br /> complained.<br /><br /> regards,tom lane<br /><br /></span></font></blockquote>
On mån, 2009-08-17 at 11:55 -0400, Alvaro Herrera wrote: > Peter Eisentraut wrote: > > > I have reworked this patch a bit and extended the plpython test suite > > around it. Current copy attached. > > I think the errcontext bits should be committed separately to get them > out of the way (and to ensure that they get in, regardless of objections > to other parts of the patch). Done that now.
On sön, 2009-08-16 at 02:44 +0300, Peter Eisentraut wrote: > The remaining problem is that the patch loses domain checking on the > return types, because some paths no longer go through the data type's > input function. I have marked these places as FIXME, and the regression > tests also contain a failing test case for this. > > What's needed here, I think, is an API that takes a datum plus type > information and checks whether the datum is valid within the domain. I > haven't found one that is exported, but maybe someone could give a tip. Got that fixed now. Updated patch is attached. I will sleep over it, but I think it's good to go.
Attachment
On mån, 2009-08-31 at 23:41 +0300, Peter Eisentraut wrote: > On sön, 2009-08-16 at 02:44 +0300, Peter Eisentraut wrote: > > The remaining problem is that the patch loses domain checking on the > > return types, because some paths no longer go through the data type's > > input function. I have marked these places as FIXME, and the regression > > tests also contain a failing test case for this. > > > > What's needed here, I think, is an API that takes a datum plus type > > information and checks whether the datum is valid within the domain. I > > haven't found one that is exported, but maybe someone could give a tip. > > Got that fixed now. Updated patch is attached. I will sleep over it, > but I think it's good to go. committed
On Wed, 2009-05-27 at 14:25 -0700, Caleb Welton wrote: > Yes, in Python >= 2.4 there is the Decimal datatype. > > However, unlike the other mappings employed by plpythonu, Decimal > requires an import statement to be in scope. adding it as already-imported module should not be hard I think that moving to saner mappings should at least be discussed and even if it is not in scope for the user-defined function body there is nothing that prevents one from using it for conversion. The Decimal _type_ needs not to be in scope for using Decimal _instances_ maybe this should/could be controlled by a GUC. btw, can we currently use funtions in setting GUC parameters ? if we can , then we could define some python environment initializing function and then do ALTER USER xxx SET pyinit = initialise_python_for_xxx() > -Caleb > > On 5/27/09 2:07 PM, "Tom Lane" <tgl@sss.pgh.pa.us> wrote: > > Peter Eisentraut <peter_e@gmx.net> writes: > > On Wednesday 27 May 2009 21:53:31 Caleb Welton wrote: > >> ... My own > >> feeling on the matter is that PyFloat is the wrong mapping > for numeric, but > >> I didn't want to muddy this patch by changing that. > > > Yeah, that one had me wondering for a while as well, but as > you say it is > > better to address that separately. > > That was making me itch as well, in my very cursory look at > the patch. > Does Python have a saner mapping for it? > > regards, tom lane > -- Hannu Krosing http://www.2ndQuadrant.com PostgreSQL Scalability and Availability Services, Consulting and Training