Thread: convert binary string to datum
How does one convert an octet string (e.g. something like a varlena structure) to a Datum? I want to create datums for use w/ heap_form_tuple in a function returning a tuple containing bytea representations of very large integers. TIA -- Ron Peterson https://www.yellowbank.com/
2007-10-12_22:22:32-0400 Ron Peterson <ron.peterson@yellowbank.com>: > How does one convert an octet string (e.g. something like a varlena > structure) to a Datum? I want to create datums for use w/ > heap_form_tuple in a function returning a tuple containing bytea > representations of very large integers. Is this a legitimate/blessed way to go about it? aval = (bytea *)palloc( len + VARHDRSZ ); VARATT_SIZEP(aval) = len + VARHDRSZ; memcpy( VARDATA(aval), myrawdata, len ); values[0] = PointerGetDatum(aval); ...etc tuple = heap_formtuple( tupdesc, values, &isNull ); -- Ron Peterson https://www.yellowbank.com/
"Ron Peterson" <ron.peterson@yellowbank.com> writes: > Is this a legitimate/blessed way to go about it? > > aval = (bytea *)palloc( len + VARHDRSZ ); > VARATT_SIZEP(aval) = len + VARHDRSZ; > memcpy( VARDATA(aval), myrawdata, len ); > values[0] = PointerGetDatum(aval); > ...etc > tuple = heap_formtuple( tupdesc, values, &isNull ); Yes, assuming that your tuple descriptor there does in fact have a varlena data type in column 1. And normally you would define your own datatype and not use bytea. Personally I'm not entirely clear why we don't just use void* for text and bytea though. Postgres 8.3 has a different macro api here though. If you want to future-proof your code you could do (put the macro definition somewhere in your private header file after including postgres.h). #ifndef SET_VARSIZE #define SET_VARSIZE(v,l) (VARATT_SIZEP(v) = (l)) #endif aval = (bytea *)palloc( len + VARHDRSZ ); SET_VARSIZE(aval, len + VARHDRSZ); memcpy( VARDATA(aval), myrawdata, len ); values[0] = PointerGetDatum(aval); ...etc tuple = heap_formtuple( tupdesc, values, &isNull ); Also, make sure you use VARSIZE to refer to the size header at all times, don't refer to it directly. And unless you mark it with storage plain always detoast it before working with an argument or anything from heap_deform_tuple. In postgres we normally put pg_detoast_datum() directly into the DatumGetFoo() and PG_GETARG_FOO_P() macros. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com
2007-10-13_01:22:06-0400 Gregory Stark <stark@enterprisedb.com>:
> "Ron Peterson" <ron.peterson@yellowbank.com> writes:
>
> > Is this a legitimate/blessed way to go about it?
> >
> > aval = (bytea *)palloc( len + VARHDRSZ );
> > VARATT_SIZEP(aval) = len + VARHDRSZ;
> > memcpy( VARDATA(aval), myrawdata, len );
> > values[0] = PointerGetDatum(aval);
> > ...etc
> > tuple = heap_formtuple( tupdesc, values, &isNull );
>
> Yes, assuming that your tuple descriptor there does in fact have a varlena
> data type in column 1.
I think that's the part I'm missing. I'm doing:
if( get_call_result_type( fcinfo, NULL, &tupdesc ) !=
TYPEFUNC_COMPOSITE )
ereport( ERROR,
( errcode( ERRCODE_FEATURE_NOT_SUPPORTED ),
errmsg( "function returning record called in context "
"that cannot accept type record" )));
BlessTupleDesc( tupdesc );
I'm not doing anything explicit to set a varlena datatype in column 1.
How would I do that?
> And normally you would define your own datatype and not use bytea.
Actually, I already have my data in a structure much like varlena. I
see PointerGetDatum basically just casts it's argument to (Datum), which
is ultimately defined as an unsigned long, I believe. I'm not
understanding the magic that PostgreSQL uses to understand whether this
is a value or a reference, and whether it's fixed or variable length
though. That must be what the tupledesc does, but I don't think I'm
setting that up right. I think. Maybe.
-Ron-
Personally I'm not entirely clear why we don't just use void* for
> text and bytea though.
>
> Postgres 8.3 has a different macro api here though. If you want to
> future-proof your code you could do (put the macro definition somewhere in
> your private header file after including postgres.h).
>
> #ifndef SET_VARSIZE
> #define SET_VARSIZE(v,l) (VARATT_SIZEP(v) = (l))
> #endif
>
> aval = (bytea *)palloc( len + VARHDRSZ );
> SET_VARSIZE(aval, len + VARHDRSZ);
> memcpy( VARDATA(aval), myrawdata, len );
> values[0] = PointerGetDatum(aval);
> ...etc
> tuple = heap_formtuple( tupdesc, values, &isNull );
>
> Also, make sure you use VARSIZE to refer to the size header at all times,
> don't refer to it directly. And unless you mark it with storage plain always
> detoast it before working with an argument or anything from heap_deform_tuple.
> In postgres we normally put pg_detoast_datum() directly into the DatumGetFoo()
> and PG_GETARG_FOO_P() macros.
>
>
> --
> Gregory Stark
> EnterpriseDB http://www.enterprisedb.com
--
Ron Peterson
https://www.yellowbank.com/
2007-10-13_08:50:56-0400 Ron Peterson <ron.peterson@yellowbank.com>: > 2007-10-13_01:22:06-0400 Gregory Stark <stark@enterprisedb.com>: > > And normally you would define your own datatype and not use bytea. > > Actually, I already have my data in a structure much like varlena. Pght, I misunderstood what you were saying. You mean create a full-blown new type. I was thinking completely internally. Yeah, that's probably a better approach - I'll do that. -- Ron Peterson https://www.yellowbank.com/
"Ron Peterson" <ron.peterson@yellowbank.com> writes: > 2007-10-13_08:50:56-0400 Ron Peterson <ron.peterson@yellowbank.com>: >> 2007-10-13_01:22:06-0400 Gregory Stark <stark@enterprisedb.com>: > >> > And normally you would define your own datatype and not use bytea. >> >> Actually, I already have my data in a structure much like varlena. > > Pght, I misunderstood what you were saying. You mean create a > full-blown new type. I was thinking completely internally. Yeah, > that's probably a better approach - I'll do that. Or you could just define new functions which operate on bytea if you're just storing binary data. I don't understand what you mean with "internally" if you're storing this in tuples? -- Gregory Stark EnterpriseDB http://www.enterprisedb.com
2007-10-13_11:12:05-0400 Gregory Stark <stark@enterprisedb.com>: > "Ron Peterson" <ron.peterson@yellowbank.com> writes: > > > 2007-10-13_08:50:56-0400 Ron Peterson <ron.peterson@yellowbank.com>: > >> 2007-10-13_01:22:06-0400 Gregory Stark <stark@enterprisedb.com>: > > > >> > And normally you would define your own datatype and not use bytea. > >> > >> Actually, I already have my data in a structure much like varlena. > > > > Pght, I misunderstood what you were saying. You mean create a > > full-blown new type. I was thinking completely internally. Yeah, > > that's probably a better approach - I'll do that. > > Or you could just define new functions which operate on bytea if you're just > storing binary data. A tuple of two or three bytea values (cryptographic keys); e.g. (bytea, bytea) > I don't understand what you mean with "internally" if you're storing > this in tuples? I thought you were talking about something like a C structure - I think I just misunderstood what you were saying. I'm still a little mixed up about just exactly what, internally, a tuple and a tupledesc are and how they are used. I think I can get where I want to go without completely figuring that out right now though... -- Ron Peterson https://www.yellowbank.com/
"Ron Peterson" <ron.peterson@yellowbank.com> writes: > I think I can get where I want to go without completely figuring that out > right now though... What are you trying to do? -- Gregory Stark EnterpriseDB http://www.enterprisedb.com
2007-10-13_13:44:33-0400 Gregory Stark <stark@enterprisedb.com>: > "Ron Peterson" <ron.peterson@yellowbank.com> writes: > > > I think I can get where I want to go without completely figuring that out > > right now though... > > What are you trying to do? I've implemented the RSA PKCS #1 v2.1 public key cryptography standard in C. Now I'm working on implementing these functions in PostgreSQL. So, for example: select * from generate_rsa_key(); would return a tuple of three very large integers: the modulus, the public exponent, and the private exponent. (Truly secure applications would do this operation on the client, and only export the public key (modulus, public exponent) to the server - but that's another matter) My first thought was to just do something like: CREATE TYPE __full_key AS ( n bytea, e bytea, d bytea ); CREATE OR REPLACE FUNCTION generate_rsa_key( ) RETURNS __full_key AS 'y_pgcrypto.so', 'y_pg_generate_rsa_keys' LANGUAGE C STRICT IMMUTABLE; Instead, I think I'll create two new types: rsa_full_key (modulus, pub exponent, priv exponent) and rsa_part_key (modulus, exponent), that will use hex as input and output for the large integers, like: n: 86161f738222dccb5b7fbb55cf8d7bf70bb71204408807427fb352ad8768f3a61124da267f9a9938b1ca5f16190c428ce0366eb841d11e99bdb93aabbf6caec42c3c0e7469fa6ebaaf12aa8b717049a753685095728ce48a4f557eaae7c00d9ff9f6f962251ebddd60f8886fde8f79f7d2fefe66d73418f7cacea079b87b204bb0cdcd3318c472222c1dcd79078fedf984cdf3f8d8feb1cba2ad034f8e1bade70d21683e1bc8baec4afc6d05fa29249a470dcba92792978268360c82fb6432d42bf50f897a1864bff7d4bdf8d86e079e37dfd282f5369f8b4674bcc4bf027cdd0ae7e88aabfee8965c7a23875ae4682a188985afb2a3cd5dcb658666cba31553 e: e12d05c431077bd41ceb79df72bad01c541ca602a8e4091efd74970490626348c21061f5d9b334844109021d6b513b0f3d245dd51fcef89a8c1c9d520d67d92e05def4bdd2d0ca08812f41f3440735ef372355b94ca72ae167cba47a04953785690cba8902f584f12f4195df2d121668b1879a27f8adab7c766d51817e7518a7fd6843e0ee5abad1e60a3968b5b54f13abbfb01c0190e01af51b7ca3d8db0ea221952138d842e376e48a47686af45c0df52caf8ad87d836cb5c26742ee95da3aafd2d0c2cf42d32a0b4f62be3cd5ecf3ff4d72cd9c977f37f09830e5ec3155c3d62a99b075fc80f21ed87744fe35833fc692dfa4bfea06f7fbe13729bbe16e17 d: 5e7501e60b6691280279689ada8a317a5c1a09ef8fab8362ea7d60f24dc36ff3c0e7d7eb0eb6266be7f854c6c59d4d76f34dc349d049a5c8980410b9fe793e94c766e7051db05d54742991649befd0a4587887d60be30a75e57ede3a2aeee9b014997ecceaaf8e00badaa2e3e106b6ab3c5e9fa1aaa8f566077f8bcf906580338b94cc4f357cc4ea1a9c3afe199e7f52c54b0514e081cc9467756aa16a6d75b38655ed80739173d99525ed05c8cef5015b1d7a2e89358ab64e335a107e5efe57d7d9644107f062243d681af084fffcbcfc84e3b13c202de98c96e0c0fdea1990d2a4e993d138dd229ee23f90536f69d1de0bafcc2cc372552eda08d74e7b0287 I know how to create a user-defined type in C, which is probably the better solution anyway; but I was initially thinking I'd just have a keygen function that output a tuple containing the integers as bytea values. I'm getting tripped trying to do that though. -- Ron Peterson https://www.yellowbank.com/
2007-10-13_14:15:06-0400 yrp001: > select * from generate_rsa_key(); Am I making this way too complicated? Do I only have to return a C string representation of three bytea values as a tuple? (I still think a bona-fide user-defined type in C is probably better.) -- Ron Peterson https://www.yellowbank.com/
"Ron Peterson" <ron.peterson@yellowbank.com> writes: > Am I making this way too complicated? Do I only have to return a C > string representation of three bytea values as a tuple? No, if you want to define a composite type and return it then you're on the right track to be using heap_form_tuple. And using a composite type is probably the right approach. Other utilities like pageinspect do use composite types for things like this. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com
"Ron Peterson" <ron.peterson@yellowbank.com> writes: > My first thought was to just do something like: > > CREATE TYPE __full_key AS ( n bytea, e bytea, d bytea ); > > CREATE OR REPLACE FUNCTION > generate_rsa_key( ) > RETURNS > __full_key Oh, incidentally you probably don't want to name your type starting with an _. Postgres names array types starting with _ so that's likely to confuse something and if not something then at least someone. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com
2007-10-13_15:22:34-0400 Gregory Stark <stark@enterprisedb.com>:
> "Ron Peterson" <ron.peterson@yellowbank.com> writes:
>
> > My first thought was to just do something like:
> >
> > CREATE TYPE __full_key AS ( n bytea, e bytea, d bytea );
> >
> > CREATE OR REPLACE FUNCTION
> > generate_rsa_key( )
> > RETURNS
> > __full_key
>
> Oh, incidentally you probably don't want to name your type starting with an _.
> Postgres names array types starting with _ so that's likely to confuse
> something and if not something then at least someone.
Thanks.
I got it working, but returning a composite type of text values, rather
than bytea. I think that's better for me anyway, because I'd like my
type's input and output functions to take hex values instead of the
crazy bytea octet representation. I ended up doing
CREATE TYPE full_key AS ( n TEXT, e TEXT, d TEXT );
-
vals = (char**)palloc( sizeof(char*) * 3 );
// convert key parts to strings
len = mpz_sizeinbase( Y_FULL_KEY_MOD(&akey), 16 ) + 1;
vals[0] = (char *)palloc( len );
gmp_snprintf( vals[0], len, "%Zx", Y_FULL_KEY_MOD(&akey) );
...etc
if( get_call_result_type( fcinfo, NULL, &td ) != TYPEFUNC_COMPOSITE ) {
ereport( ERROR,
( errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg( "function returning record called in context "
"that cannot accept type record" )));
PG_RETURN_NULL();
}
// Make a persistant copy.
td = CreateTupleDescCopy( td );
aim = TupleDescGetAttInMetadata( td );
ht = BuildTupleFromCStrings( aim, vals );
/* make the tuple into a datum */
result = HeapTupleGetDatum( ht );
Someday I'd still like to figure out how to return a composite type
containing bytea values...
--
Ron Peterson
https://www.yellowbank.com/