Thread: UDFs

UDFs

From
jf
Date:
Hi.

Trying to implement some simple digest routines via UDFs and for whatever
reason I get: ERROR:  invalid memory alloc request size 4294967293 on
PG_RETURN_TEXT_P(); any ideas what the issue is exactly?

The code is verified as working when pulled out of a UDF and put into a
normal C program. Thanks.

dp_blag=# CREATE OR REPLACE FUNCTION sha512(text) RETURNS text AS
'dp_sha512.so','dp_sha512' LANGUAGE C STRICT;
CREATE FUNCTION
dp_blag=# SELECT sha512('jf');
ERROR:  invalid memory alloc request size 4294967293

PG_FUNCTION_INFO_V1(dp_sha512);

Datum
dp_sha512(PG_FUNCTION_ARGS)
{
        text *          hash;
        text *          plain;
        int32           hlen;

        plain = (text *)PG_GETARG_TEXT_P(0);

        if (NULL == plain) {
                ereport(ERROR,
            (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED),
            errmsg("invalid argument to function"))
        );
                PG_RETURN_NULL();
        }

        hlen = SHA512_DIGEST_LENGTH;

        hash = (text *)palloc(hlen+1);

        if (NULL == hash) {
                ereport(ERROR,
            (errcode(ERRCODE_INTERNAL_ERROR),
            errmsg("palloc() failed"))
        );
                PG_RETURN_NULL();
        }

        memset(VARDATA(hash), 0, hlen);
        SHA512(VARDATA(plain), hlen, VARDATA(hash));
        PG_RETURN_TEXT_P(hash);
}

Re: UDFs

From
"Pavel Stehule"
Date:
>
>         hash = (text *)palloc(hlen+1);

??? palloc(hlen + VARHDRSZ)

>
>         memset(VARDATA(hash), 0, hlen);
>         SHA512(VARDATA(plain), hlen, VARDATA(hash));


++ VARATT_SIZEP (hash) = VARHDRSZ + ....;

>         PG_RETURN_TEXT_P(hash);
> }
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
>                http://www.postgresql.org/docs/faq
>

Re: UDFs

From
Martijn van Oosterhout
Date:
On Mon, Aug 13, 2007 at 03:17:36PM +0000, jf wrote:
> Hi.
>
> Trying to implement some simple digest routines via UDFs and for whatever
> reason I get: ERROR:  invalid memory alloc request size 4294967293 on
> PG_RETURN_TEXT_P(); any ideas what the issue is exactly?

A few points about your code:
- The tests against NULL are useless: palloc never returns NULL and
the argument to the function won't be NULL either (it's decalred
STRICT).
- Your palloc should be for more, the header may be more than one byte
VAR_HEADER_LEN or some such would be more appropriate)
- Finally, what is probably the actual problem, at no point did you
assign a length to the hash variable, ie VARLEN(hash)=foo.

Hope this helps,
--
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.

Attachment

Re: UDFs

From
jf
Date:
Yup, an uninitialized variable makes sense, That appears to have worked, I appreciate it. In regards to the checks for
NULL, I wasn't sure if it was necessary or not so I opted for the safe
route, thanks for clearing that up for me.

The output from the function is binary, does it matter if I use text or
bytea? Finally, how do I cast in SQL? when I try to pass the output to
encode() it says it can't find a function taking those types.

Thanks a lot!

On Mon, 13 Aug 2007, Martijn van Oosterhout wrote:

> Date: Mon, 13 Aug 2007 10:42:52 +0200
> From: Martijn van Oosterhout <kleptog@svana.org>
> To: jf <jf@danglingpointers.net>
> Cc: pgsql-general@postgresql.org
> Subject: Re: [GENERAL] UDFs
>
> On Mon, Aug 13, 2007 at 03:17:36PM +0000, jf wrote:
> > Hi.
> >
> > Trying to implement some simple digest routines via UDFs and for whatever
> > reason I get: ERROR:  invalid memory alloc request size 4294967293 on
> > PG_RETURN_TEXT_P(); any ideas what the issue is exactly?
>
> A few points about your code:
> - The tests against NULL are useless: palloc never returns NULL and
> the argument to the function won't be NULL either (it's decalred
> STRICT).
> - Your palloc should be for more, the header may be more than one byte
> VAR_HEADER_LEN or some such would be more appropriate)
> - Finally, what is probably the actual problem, at no point did you
> assign a length to the hash variable, ie VARLEN(hash)=foo.
>
> Hope this helps,
>

Re: UDFs

From
hubert depesz lubaczewski
Date:
On Mon, Aug 13, 2007 at 03:17:36PM +0000, jf wrote:
> Trying to implement some simple digest routines via UDFs and for whatever
> reason I get: ERROR:  invalid memory alloc request size 4294967293 on
> PG_RETURN_TEXT_P(); any ideas what the issue is exactly?

just checking - you do realize that it's reinventing the wheel?
i mean - the code already exists, and is bundled in standard postgresql
sources?
if not - use pgcrypto module from contrib/

depesz

--
quicksil1er: "postgres is excellent, but like any DB it requires a
highly paid DBA.  here's my CV!" :)
http://www.depesz.com/ - blog dla ciebie (i moje CV)

Re: UDFs

From
jf
Date:
my understanding was that pgcrypto was not compiled by default?
Furthermore, finding next to no documentation about it online and deciding
I only needed one function instead of an entire crypto API i decided it
would make the most sense to just code the 10 lines to do it myself.


 On Mon, 13 Aug 2007, hubert depesz lubaczewski wrote:

> Date: Mon, 13 Aug 2007 11:29:58 +0200
> From: hubert depesz lubaczewski <depesz@depesz.com>
> To: jf <jf@danglingpointers.net>
> Cc: pgsql-general@postgresql.org
> Subject: Re: [GENERAL] UDFs
>
> On Mon, Aug 13, 2007 at 03:17:36PM +0000, jf wrote:
> > Trying to implement some simple digest routines via UDFs and for whatever
> > reason I get: ERROR:  invalid memory alloc request size 4294967293 on
> > PG_RETURN_TEXT_P(); any ideas what the issue is exactly?
>
> just checking - you do realize that it's reinventing the wheel?
> i mean - the code already exists, and is bundled in standard postgresql
> sources?
> if not - use pgcrypto module from contrib/
>
> depesz
>
>

Re: UDFs

From
hubert depesz lubaczewski
Date:
On Mon, Aug 13, 2007 at 05:07:50PM +0000, jf wrote:
> my understanding was that pgcrypto was not compiled by default?

your function is also not compiled on default. but pgcrypto is at the
very least available by default (in sources, or in precompiled
packages).

> Furthermore, finding next to no documentation about it online and deciding

i dont know about online documentation, but in installed pgcrypto you
have a nice readme with all functions described.

> I only needed one function instead of an entire crypto API i decided it
> would make the most sense to just code the 10 lines to do it myself.

i'm not opposing to the idea. i was just checking if you do know about
it :)

best regards,

depesz

--
quicksil1er: "postgres is excellent, but like any DB it requires a
highly paid DBA.  here's my CV!" :)
http://www.depesz.com/ - blog dla ciebie (i moje CV)

Re: UDFs

From
jf
Date:
>
> your function is also not compiled on default. but pgcrypto is at the
> very least available by default (in sources, or in precompiled
> packages).

Yes I understand, and trust me I'm typically not a 'not made in my house'
type, but google for pgcrypto, you don't find much out there other than
'look in contrib/', and honestly I wasn't about to sit down and recompile
everything to get a bunch of functionality I really didn't need, I (in
hindsight incorrectly) decided it would be quicker just to write a UDF.

Where the incorrectly part is that I've been battling for a couple hours
trying to figure out why even though I had changed the UDF to use BYTEA
primitives instead of TEXT it was still using TEXTs, and I just learned
something new, CREATE OR REPLACE FUNCTION [...] doesn't replace the
function if you change the types; it makes sense, I've just trying to
figure out wth is going on.

> i dont know about online documentation, but in installed pgcrypto you
> have a nice readme with all functions described.

Expecting people who may or may not have the source sitting around anymore
to download the source to look at the documentation to even determine if
they want to use that code is not a very effective method of marketing,
imho of course.

This holds especially true when the online manual for the
DB itself is lacking in many places (ie can palloc() fail? does palloc0()
zero out the memory returned? et cetera), I had to dig through the header
files just to find the various types and such, and then its still not
entirely clear what exactly the header for the variable length types is,
in one header file it gets typedef'd to a structure defined right above it
that contains a length field and an array of a single character (does that
serve as a pointer to the beginning of the data?), but then the
VARDATA()/VARSIZE()/et cetera macro's appear to work on an entirely
different structure, but there is no immediate correlation between the two
structures in the headers.

It's a bit sad because the documentation for pgcrypto is actually quite a
bit better than for the UDFs, and it wasn't until after I had made the
decision that the UDFs would be easier and fought with it for some time
that I realized this.


> i'm not opposing to the idea. i was just checking if you do know about
> it :)

understood, I appreciate the suggestion. In addition I couldn't find any
documentation that told me how to install the functions in pgcrypto (do I
need to CREATE FUNCTION for every function in there?), the README mentions
a .sql file thats supposed to get you up and going, but the only .sql file
I found in the directory was an uninstall .sql file, am I expected to grok
it and do the inverse for every step? et cetera.

I don't mean to come off as a complainer, but the documentation for the DB
overall seems to be lacking, and that was my main reason for opting out,
even if now that all is said and done it would've been easier to use
pgcrypto.

 > best regards,
>
> depesz


cheers,

jf

Re: UDFs

From
hubert depesz lubaczewski
Date:
On Mon, Aug 13, 2007 at 07:22:26PM +0000, jf wrote:
> understood, I appreciate the suggestion. In addition I couldn't find any
> documentation that told me how to install the functions in pgcrypto (do I
> need to CREATE FUNCTION for every function in there?), the README mentions
> a .sql file thats supposed to get you up and going, but the only .sql file
> I found in the directory was an uninstall .sql file, am I expected to grok
> it and do the inverse for every step? et cetera.
>

1. as for installing contrib - usually when you install psotgresql from
prebuilt binary packages, there is also contrib package. for example on
ubuntu it is postgresql-contrib-8.2, so simple: apt-get install
postgresql-contrib-8.2 will install it.

as for installation in sql - for every contrib module you get sql
script. it is usually called <contrib_module>.sql (pgcrypto.sql).

usually it's in: <PREFIX>/share/postgresql/contrib/pgcrypto.sql

in the database you want to use pgcrypto functions, you simply run this
sql (as superuser), and that's all.

depesz

--
quicksil1er: "postgres is excellent, but like any DB it requires a
highly paid DBA.  here's my CV!" :)
http://www.depesz.com/ - blog dla ciebie (i moje CV)

Re: UDFs

From
jf
Date:
> 1. as for installing contrib - usually when you install psotgresql from
> prebuilt binary packages, there is also contrib package. for example on
> ubuntu it is postgresql-contrib-8.2, so simple: apt-get install
> postgresql-contrib-8.2 will install it.

I actually built from source, and just didnt know that crypto wasnt
included until I was already up and running and had gotten far enough into
things that I didn't want to restart.

> as for installation in sql - for every contrib module you get sql
> script. it is usually called <contrib_module>.sql (pgcrypto.sql).
>
> usually it's in: <PREFIX>/share/postgresql/contrib/pgcrypto.sql
> in the database you want to use pgcrypto functions, you simply run this
> sql (as superuser), and that's all.

theory# pwd
/home/jf/postgresql-8.2.4
theory# cd share
bash: cd: share: No such file or directory
theory# pwd
/home/jf/postgresql-8.2.4
theory# ls share
ls: cannot access share: No such file or directory
theory# find ./ -name pgcrypto.sql
theory# cd contrib/pgcrypto/
theory# ls *.sql
uninstall_pgcrypto.sql
theory# ls
Makefile         crypt-blowfish.c  expected   imath.h          mbuf.h
pgcrypto.c       pgp-cfb.c       pgp-info.c          pgp-pgsql.c
pgp-s2k.c   px-crypt.h  random.c      sha1.c  sql
README.pgcrypto  crypt-des.c       fortuna.c  internal-sha2.c  md5.c
pgcrypto.h       pgp-compress.c  pgp-mpi-internal.c  pgp-pubdec.c  pgp.c
px-hmac.c   rijndael.c    sha1.h  uninstall_pgcrypto.sql
blf.c            crypt-gensalt.c   fortuna.h  internal.c       md5.h
pgcrypto.sql.in  pgp-decrypt.c   pgp-mpi-openssl.c   pgp-pubenc.c  pgp.h
px.c        rijndael.h    sha2.c
blf.h            crypt-md5.c       imath.c    mbuf.c           openssl.c
pgp-armor.c      pgp-encrypt.c   pgp-mpi.c           pgp-pubkey.c
px-crypt.c  px.h        rijndael.tbl  sha2.h


Re: UDFs

From
hubert depesz lubaczewski
Date:
On Mon, Aug 13, 2007 at 07:36:41PM +0000, jf wrote:
> > usually it's in: <PREFIX>/share/postgresql/contrib/pgcrypto.sql
> > in the database you want to use pgcrypto functions, you simply run this
> > sql (as superuser), and that's all.
> theory# pwd
> /home/jf/postgresql-8.2.4
> theory# cd share
> bash: cd: share: No such file or directory
> theory# pwd
> /home/jf/postgresql-8.2.4
> theory# ls share
> ls: cannot access share: No such file or directory

share of *installed* system. if you compiled with --prefix=/usr/local,
then it would be /usr/local/share/postgresql/...

> theory# find ./ -name pgcrypto.sql
> theory# cd contrib/pgcrypto/
> theory# ls *.sql

if you didn't compile it - it is not built.
simply:
cd contrib/pgcrypto
make
make install
and you should be fine.

i usually compile postgresql in this way:
cd postgre*
./configure --.... some options
make
make install
cd contrib
make
make install

in this way i have all contrib modules compiled.

of course - you dont need all (on the other hand - i strongly suggest
that you get some familiarity with it, as there are some real gems).

depesz

--
quicksil1er: "postgres is excellent, but like any DB it requires a
highly paid DBA.  here's my CV!" :)
http://www.depesz.com/ - blog dla ciebie (i moje CV)

Re: UDFs

From
jf
Date:
> share of *installed* system. if you compiled with --prefix=/usr/local,
> then it would be /usr/local/share/postgresql/...

Ah you have to forgive me, I'm in the states and its quite late ;]

> of course - you dont need all (on the other hand - i strongly suggest
> that you get some familiarity with it, as there are some real gems).

Well as I continue working on my backend and find that I need some other
functionality, I'll check there first ;]

Thanks again.

Re: UDFs

From
Martijn van Oosterhout
Date:
On Mon, Aug 13, 2007 at 07:36:41PM +0000, jf wrote:
> > usually it's in: <PREFIX>/share/postgresql/contrib/pgcrypto.sql
> > in the database you want to use pgcrypto functions, you simply run this
> > sql (as superuser), and that's all.
>
> theory# pwd
> /home/jf/postgresql-8.2.4
> theory# cd share
> bash: cd: share: No such file or directory

He means the INSTALLPREFIX, which is usually /usr or /usr/local. Your
problem is that you havn't build it yet (run make) which will generate
the actual module and you need to install it to get the scripts into
the right locations.

If you can, I'd suggest installing binary versions of the contrib
modules (it's called postgresql-8.2-contrib in debian for example).
That saves you having to worry about sources, paths, compilations, etc).

Have a nice day,
--
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.

Attachment