Thread: UDFs
Hi.
Trying to implement some simple digest routines via UDFs and for whatever
reason I get: ERROR: invalid memory alloc request size 4294967293 on
PG_RETURN_TEXT_P(); any ideas what the issue is exactly?
The code is verified as working when pulled out of a UDF and put into a
normal C program. Thanks.
dp_blag=# CREATE OR REPLACE FUNCTION sha512(text) RETURNS text AS
'dp_sha512.so','dp_sha512' LANGUAGE C STRICT;
CREATE FUNCTION
dp_blag=# SELECT sha512('jf');
ERROR: invalid memory alloc request size 4294967293
PG_FUNCTION_INFO_V1(dp_sha512);
Datum
dp_sha512(PG_FUNCTION_ARGS)
{
text * hash;
text * plain;
int32 hlen;
plain = (text *)PG_GETARG_TEXT_P(0);
if (NULL == plain) {
ereport(ERROR,
(errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED),
errmsg("invalid argument to function"))
);
PG_RETURN_NULL();
}
hlen = SHA512_DIGEST_LENGTH;
hash = (text *)palloc(hlen+1);
if (NULL == hash) {
ereport(ERROR,
(errcode(ERRCODE_INTERNAL_ERROR),
errmsg("palloc() failed"))
);
PG_RETURN_NULL();
}
memset(VARDATA(hash), 0, hlen);
SHA512(VARDATA(plain), hlen, VARDATA(hash));
PG_RETURN_TEXT_P(hash);
}
> > hash = (text *)palloc(hlen+1); ??? palloc(hlen + VARHDRSZ) > > memset(VARDATA(hash), 0, hlen); > SHA512(VARDATA(plain), hlen, VARDATA(hash)); ++ VARATT_SIZEP (hash) = VARHDRSZ + ....; > PG_RETURN_TEXT_P(hash); > } > > ---------------------------(end of broadcast)--------------------------- > TIP 3: Have you checked our extensive FAQ? > > http://www.postgresql.org/docs/faq >
On Mon, Aug 13, 2007 at 03:17:36PM +0000, jf wrote: > Hi. > > Trying to implement some simple digest routines via UDFs and for whatever > reason I get: ERROR: invalid memory alloc request size 4294967293 on > PG_RETURN_TEXT_P(); any ideas what the issue is exactly? A few points about your code: - The tests against NULL are useless: palloc never returns NULL and the argument to the function won't be NULL either (it's decalred STRICT). - Your palloc should be for more, the header may be more than one byte VAR_HEADER_LEN or some such would be more appropriate) - Finally, what is probably the actual problem, at no point did you assign a length to the hash variable, ie VARLEN(hash)=foo. Hope this helps, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to litigate.
Attachment
Yup, an uninitialized variable makes sense, That appears to have worked, I appreciate it. In regards to the checks for NULL, I wasn't sure if it was necessary or not so I opted for the safe route, thanks for clearing that up for me. The output from the function is binary, does it matter if I use text or bytea? Finally, how do I cast in SQL? when I try to pass the output to encode() it says it can't find a function taking those types. Thanks a lot! On Mon, 13 Aug 2007, Martijn van Oosterhout wrote: > Date: Mon, 13 Aug 2007 10:42:52 +0200 > From: Martijn van Oosterhout <kleptog@svana.org> > To: jf <jf@danglingpointers.net> > Cc: pgsql-general@postgresql.org > Subject: Re: [GENERAL] UDFs > > On Mon, Aug 13, 2007 at 03:17:36PM +0000, jf wrote: > > Hi. > > > > Trying to implement some simple digest routines via UDFs and for whatever > > reason I get: ERROR: invalid memory alloc request size 4294967293 on > > PG_RETURN_TEXT_P(); any ideas what the issue is exactly? > > A few points about your code: > - The tests against NULL are useless: palloc never returns NULL and > the argument to the function won't be NULL either (it's decalred > STRICT). > - Your palloc should be for more, the header may be more than one byte > VAR_HEADER_LEN or some such would be more appropriate) > - Finally, what is probably the actual problem, at no point did you > assign a length to the hash variable, ie VARLEN(hash)=foo. > > Hope this helps, >
On Mon, Aug 13, 2007 at 03:17:36PM +0000, jf wrote: > Trying to implement some simple digest routines via UDFs and for whatever > reason I get: ERROR: invalid memory alloc request size 4294967293 on > PG_RETURN_TEXT_P(); any ideas what the issue is exactly? just checking - you do realize that it's reinventing the wheel? i mean - the code already exists, and is bundled in standard postgresql sources? if not - use pgcrypto module from contrib/ depesz -- quicksil1er: "postgres is excellent, but like any DB it requires a highly paid DBA. here's my CV!" :) http://www.depesz.com/ - blog dla ciebie (i moje CV)
my understanding was that pgcrypto was not compiled by default? Furthermore, finding next to no documentation about it online and deciding I only needed one function instead of an entire crypto API i decided it would make the most sense to just code the 10 lines to do it myself. On Mon, 13 Aug 2007, hubert depesz lubaczewski wrote: > Date: Mon, 13 Aug 2007 11:29:58 +0200 > From: hubert depesz lubaczewski <depesz@depesz.com> > To: jf <jf@danglingpointers.net> > Cc: pgsql-general@postgresql.org > Subject: Re: [GENERAL] UDFs > > On Mon, Aug 13, 2007 at 03:17:36PM +0000, jf wrote: > > Trying to implement some simple digest routines via UDFs and for whatever > > reason I get: ERROR: invalid memory alloc request size 4294967293 on > > PG_RETURN_TEXT_P(); any ideas what the issue is exactly? > > just checking - you do realize that it's reinventing the wheel? > i mean - the code already exists, and is bundled in standard postgresql > sources? > if not - use pgcrypto module from contrib/ > > depesz > >
On Mon, Aug 13, 2007 at 05:07:50PM +0000, jf wrote: > my understanding was that pgcrypto was not compiled by default? your function is also not compiled on default. but pgcrypto is at the very least available by default (in sources, or in precompiled packages). > Furthermore, finding next to no documentation about it online and deciding i dont know about online documentation, but in installed pgcrypto you have a nice readme with all functions described. > I only needed one function instead of an entire crypto API i decided it > would make the most sense to just code the 10 lines to do it myself. i'm not opposing to the idea. i was just checking if you do know about it :) best regards, depesz -- quicksil1er: "postgres is excellent, but like any DB it requires a highly paid DBA. here's my CV!" :) http://www.depesz.com/ - blog dla ciebie (i moje CV)
> > your function is also not compiled on default. but pgcrypto is at the > very least available by default (in sources, or in precompiled > packages). Yes I understand, and trust me I'm typically not a 'not made in my house' type, but google for pgcrypto, you don't find much out there other than 'look in contrib/', and honestly I wasn't about to sit down and recompile everything to get a bunch of functionality I really didn't need, I (in hindsight incorrectly) decided it would be quicker just to write a UDF. Where the incorrectly part is that I've been battling for a couple hours trying to figure out why even though I had changed the UDF to use BYTEA primitives instead of TEXT it was still using TEXTs, and I just learned something new, CREATE OR REPLACE FUNCTION [...] doesn't replace the function if you change the types; it makes sense, I've just trying to figure out wth is going on. > i dont know about online documentation, but in installed pgcrypto you > have a nice readme with all functions described. Expecting people who may or may not have the source sitting around anymore to download the source to look at the documentation to even determine if they want to use that code is not a very effective method of marketing, imho of course. This holds especially true when the online manual for the DB itself is lacking in many places (ie can palloc() fail? does palloc0() zero out the memory returned? et cetera), I had to dig through the header files just to find the various types and such, and then its still not entirely clear what exactly the header for the variable length types is, in one header file it gets typedef'd to a structure defined right above it that contains a length field and an array of a single character (does that serve as a pointer to the beginning of the data?), but then the VARDATA()/VARSIZE()/et cetera macro's appear to work on an entirely different structure, but there is no immediate correlation between the two structures in the headers. It's a bit sad because the documentation for pgcrypto is actually quite a bit better than for the UDFs, and it wasn't until after I had made the decision that the UDFs would be easier and fought with it for some time that I realized this. > i'm not opposing to the idea. i was just checking if you do know about > it :) understood, I appreciate the suggestion. In addition I couldn't find any documentation that told me how to install the functions in pgcrypto (do I need to CREATE FUNCTION for every function in there?), the README mentions a .sql file thats supposed to get you up and going, but the only .sql file I found in the directory was an uninstall .sql file, am I expected to grok it and do the inverse for every step? et cetera. I don't mean to come off as a complainer, but the documentation for the DB overall seems to be lacking, and that was my main reason for opting out, even if now that all is said and done it would've been easier to use pgcrypto. > best regards, > > depesz cheers, jf
On Mon, Aug 13, 2007 at 07:22:26PM +0000, jf wrote: > understood, I appreciate the suggestion. In addition I couldn't find any > documentation that told me how to install the functions in pgcrypto (do I > need to CREATE FUNCTION for every function in there?), the README mentions > a .sql file thats supposed to get you up and going, but the only .sql file > I found in the directory was an uninstall .sql file, am I expected to grok > it and do the inverse for every step? et cetera. > 1. as for installing contrib - usually when you install psotgresql from prebuilt binary packages, there is also contrib package. for example on ubuntu it is postgresql-contrib-8.2, so simple: apt-get install postgresql-contrib-8.2 will install it. as for installation in sql - for every contrib module you get sql script. it is usually called <contrib_module>.sql (pgcrypto.sql). usually it's in: <PREFIX>/share/postgresql/contrib/pgcrypto.sql in the database you want to use pgcrypto functions, you simply run this sql (as superuser), and that's all. depesz -- quicksil1er: "postgres is excellent, but like any DB it requires a highly paid DBA. here's my CV!" :) http://www.depesz.com/ - blog dla ciebie (i moje CV)
> 1. as for installing contrib - usually when you install psotgresql from > prebuilt binary packages, there is also contrib package. for example on > ubuntu it is postgresql-contrib-8.2, so simple: apt-get install > postgresql-contrib-8.2 will install it. I actually built from source, and just didnt know that crypto wasnt included until I was already up and running and had gotten far enough into things that I didn't want to restart. > as for installation in sql - for every contrib module you get sql > script. it is usually called <contrib_module>.sql (pgcrypto.sql). > > usually it's in: <PREFIX>/share/postgresql/contrib/pgcrypto.sql > in the database you want to use pgcrypto functions, you simply run this > sql (as superuser), and that's all. theory# pwd /home/jf/postgresql-8.2.4 theory# cd share bash: cd: share: No such file or directory theory# pwd /home/jf/postgresql-8.2.4 theory# ls share ls: cannot access share: No such file or directory theory# find ./ -name pgcrypto.sql theory# cd contrib/pgcrypto/ theory# ls *.sql uninstall_pgcrypto.sql theory# ls Makefile crypt-blowfish.c expected imath.h mbuf.h pgcrypto.c pgp-cfb.c pgp-info.c pgp-pgsql.c pgp-s2k.c px-crypt.h random.c sha1.c sql README.pgcrypto crypt-des.c fortuna.c internal-sha2.c md5.c pgcrypto.h pgp-compress.c pgp-mpi-internal.c pgp-pubdec.c pgp.c px-hmac.c rijndael.c sha1.h uninstall_pgcrypto.sql blf.c crypt-gensalt.c fortuna.h internal.c md5.h pgcrypto.sql.in pgp-decrypt.c pgp-mpi-openssl.c pgp-pubenc.c pgp.h px.c rijndael.h sha2.c blf.h crypt-md5.c imath.c mbuf.c openssl.c pgp-armor.c pgp-encrypt.c pgp-mpi.c pgp-pubkey.c px-crypt.c px.h rijndael.tbl sha2.h
On Mon, Aug 13, 2007 at 07:36:41PM +0000, jf wrote: > > usually it's in: <PREFIX>/share/postgresql/contrib/pgcrypto.sql > > in the database you want to use pgcrypto functions, you simply run this > > sql (as superuser), and that's all. > theory# pwd > /home/jf/postgresql-8.2.4 > theory# cd share > bash: cd: share: No such file or directory > theory# pwd > /home/jf/postgresql-8.2.4 > theory# ls share > ls: cannot access share: No such file or directory share of *installed* system. if you compiled with --prefix=/usr/local, then it would be /usr/local/share/postgresql/... > theory# find ./ -name pgcrypto.sql > theory# cd contrib/pgcrypto/ > theory# ls *.sql if you didn't compile it - it is not built. simply: cd contrib/pgcrypto make make install and you should be fine. i usually compile postgresql in this way: cd postgre* ./configure --.... some options make make install cd contrib make make install in this way i have all contrib modules compiled. of course - you dont need all (on the other hand - i strongly suggest that you get some familiarity with it, as there are some real gems). depesz -- quicksil1er: "postgres is excellent, but like any DB it requires a highly paid DBA. here's my CV!" :) http://www.depesz.com/ - blog dla ciebie (i moje CV)
> share of *installed* system. if you compiled with --prefix=/usr/local, > then it would be /usr/local/share/postgresql/... Ah you have to forgive me, I'm in the states and its quite late ;] > of course - you dont need all (on the other hand - i strongly suggest > that you get some familiarity with it, as there are some real gems). Well as I continue working on my backend and find that I need some other functionality, I'll check there first ;] Thanks again.
On Mon, Aug 13, 2007 at 07:36:41PM +0000, jf wrote: > > usually it's in: <PREFIX>/share/postgresql/contrib/pgcrypto.sql > > in the database you want to use pgcrypto functions, you simply run this > > sql (as superuser), and that's all. > > theory# pwd > /home/jf/postgresql-8.2.4 > theory# cd share > bash: cd: share: No such file or directory He means the INSTALLPREFIX, which is usually /usr or /usr/local. Your problem is that you havn't build it yet (run make) which will generate the actual module and you need to install it to get the scripts into the right locations. If you can, I'd suggest installing binary versions of the contrib modules (it's called postgresql-8.2-contrib in debian for example). That saves you having to worry about sources, paths, compilations, etc). Have a nice day, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to litigate.