Thread: ECPG: Automatic Storage allocation for NULL-pointing output variables

Hi there,

I'm looking for an alternative to statically sized character arrays
within ECPG.

E.g. exec sql select relname into :VAR from pg_classes;

It should (!) be possible to obtain this by using pointers which are
initialized to NULL.

It is possible for   integer *var=0;       reading an arbitrary number of integers into a malloced array
and   char *var=0;       reading one arbitrary length string into a malloced character
array

Works perfectly.

But ecpg does not accept   char **var=0;

Is there any way to specify arbitrary strings of any length ?

The code (execute.c:435) has also support for varchar - but how to
declare such a variable length varchar?

And the code (execute.c:432) multiplies the number of tuples with the
longest string. But how to use this code in real life?

Looks like dead code to me (hopefully I'm wrong)

Yours   Christof

PS: I'm willing to implement char**
(allocate tuples*sizeof(char*) + sum(length(tuple[n])+1) bytes,fill the array with pointers,fill the space behind the
arraywith the actual data.This means just one free get's it all.But this code cannot be portable!)
 

On Wed, Oct 24, 2001 at 01:27:48PM +0200, Christof Petig wrote:
> But ecpg does not accept
>     char **var=0;

You mean the parser I guess. Well, I'm not even sure an array of strings
ever has been implemented.
> Is there any way to specify arbitrary strings of any length ?

char *var=NULL; declares an empty pointer which is then allocated by
libecpg.

> The code (execute.c:435) has also support for varchar - but how to
> declare such a variable length varchar?

Yes, as ecpg says:

ERROR: pointer to varchar are not implemented

But you should be able to work around this by using something like this:

exec sql type str is varchar[8];
...
str *var=NULL;
...

However, I did not try this, so I'm not sure.

> And the code (execute.c:432) multiplies the number of tuples with the
> longest string. But how to use this code in real life?

Sorry, I do not understand that. Where's the problem? The code is executed
after the result has been read using libpq. So it already knows all the
result strings and can calculate the longest one.

> PS: I'm willing to implement char**
> (allocate tuples*sizeof(char*) + sum(length(tuple[n])+1) bytes,

Or as it is now tuples*sizeof(char*)*max(length(tuple[n])+1) bytes.

>  fill the array with pointers,
>  fill the space behind the array with the actual data.
>  This means just one free get's it all.

Which indeed is important.

>  But this code cannot be portable!)

Which is exactly the problem.

Michael
-- 
Michael Meskes
Michael@Fam-Meskes.De
Go SF 49ers! Go Rhein Fire!
Use Debian GNU/Linux! Use PostgreSQL!

Michael Meskes wrote:

> On Wed, Oct 24, 2001 at 01:27:48PM +0200, Christof Petig wrote:
> > But ecpg does not accept
> >     char **var=0;
>
> You mean the parser I guess. Well, I'm not even sure an array of strings
> ever has been implemented.

I think it has never been implemented.

> > Is there any way to specify arbitrary strings of any length ?

Actually I'm looking for a dynamic length array of dynamically allocated
strings. (Sorry for my English, in German it's [beliebig viele Strings von
beliebiger Länge] to make it clear to you?).

If you consider how libepcg  is implemented now, if you want to avoid network
traffic (and latency) you should transfer all results at once. And you can
read these results into an array of host variables.

But if you don't know (or don't want to hard code) how many rows are possible
(maximum), you cannot use an preallocated array. With int's it's fine [int
*var=0;] but the text types are the problem.

> char *var=NULL; declares an empty pointer which is then allocated by
> libecpg.

Sorry, I need a multi  string array.

> > The code (execute.c:435) has also support for varchar - but how to
> > declare such a variable length varchar?
> exec sql type str is varchar[8];
> str *var=NULL;

Shudder. There should be a decent syntax to specify this. If anyone ever
needs this functionality!!!

Propose _if_ ever needed:
What about using a different varchar type e.g. [struct varchar_vca  { int
len; char *arr; }] which corresponds to  varchar   or   varchar[]  in the
parser. But the user should remember to free it afterwards. Sadly we can't
default to C++ (destructors).

> > And the code (execute.c:432) multiplies the number of tuples with the
> > longest string. But how to use this code in real life?

> Sorry, I do not understand that. Where's the problem? The code is executed
> after the result has been read using libpq. So it already knows all the
> result strings and can calculate the longest one.

How to declare a variable which uses this functionality?

> > PS: I'm willing to implement char**
> > (allocate tuples*sizeof(char*) + sum(length(tuple[n])+1) bytes,
>
> Or as it is now tuples*sizeof(char*)*max(length(tuple[n])+1) bytes.

Yes, but which kind of host variable do you propose to get this feature?
That's why I talked about dead code. I couldn't declare a host variable which
used this feature.

> >  But this code cannot be portable!)
>
> Which is exactly the problem.

At first embedded SQL was meant to be portable. And we successfully ported
our LNX and Adabas D programs to Postgres. And we share applications between
Oracle and Postgres databases.

But I don't remember reading anything about dynamic allocation of host
variables in the standard. And I don't believe any longer in the
meaningfullness of this standard (after I read the oracle docs about _their_
style of embedded sql).

And ... I don't believe that the standard is this clever to foresee my
proposal for this problem. So using this vital feature makes your program
non-standard.

Christof

On Fri, Oct 26, 2001 at 09:49:47AM +0200, Christof Petig wrote:
> I think it has never been implemented.

Yes, if my memory serves well, I wasn't sure how to implement this so it
works on all architectures.

> Actually I'm looking for a dynamic length array of dynamically allocated
> strings. (Sorry for my English, in German it's [beliebig viele Strings von
> beliebiger Länge] to make it clear to you?).

Sure. That's the exact problem because this is not a two dimensional array
of characters but an array of pointers.

> If you consider how libepcg  is implemented now, if you want to avoid network
> traffic (and latency) you should transfer all results at once. And you can
> read these results into an array of host variables.
> ...

Sure. That's why I implemented it. :-)

> > exec sql type str is varchar[8];
> > str *var=NULL;
> 
> Shudder. There should be a decent syntax to specify this. If anyone ever
> needs this functionality!!!

You're free to add this. It's not trivial though.

> Propose _if_ ever needed:
> What about using a different varchar type e.g. [struct varchar_vca  { int
> len; char *arr; }] which corresponds to  varchar   or   varchar[]  in the
> parser. But the user should remember to free it afterwards. Sadly we can't
> default to C++ (destructors).

How should that work? The use only defines a variable as varchar. The rest
is doen by ecpg. In fact I would prefer if the user does not have to use
var.arr to access the string but we could find a way to make this completely
transparent.

> > Sorry, I do not understand that. Where's the problem? The code is executed
> > after the result has been read using libpq. So it already knows all the
> > result strings and can calculate the longest one.
> 
> How to declare a variable which uses this functionality?

I see. Yes, that's the problem. The very same code works with all datatypes
except char *.

> Yes, but which kind of host variable do you propose to get this feature?
> That's why I talked about dead code. I couldn't declare a host variable which
> used this feature.

That's true. 

> At first embedded SQL was meant to be portable. And we successfully ported
> our LNX and Adabas D programs to Postgres. And we share applications between
> Oracle and Postgres databases.
> ...

Don't get me wrong. I did not mean portable to other DBMSs but portable to
other OSs and C compiler. If it works on all platforms PostgreSQL works on,
I'm satisfied.

Michael
-- 
Michael Meskes
Michael@Fam-Meskes.De
Go SF 49ers! Go Rhein Fire!
Use Debian GNU/Linux! Use PostgreSQL!

Michael Meskes wrote:

> > Actually I'm looking for a dynamic length array of dynamically allocated
> > strings. (Sorry for my English, in German it's [beliebig viele Strings von
> > beliebiger Länge] to make it clear to you?).
>
> Sure. That's the exact problem because this is not a two dimensional array
> of characters but an array of pointers.

right. I'll try to implement it (since this looks like the most promising way to
me). And I would love to have this functionality in 7.2, fast.

So char **var = NULL;

get's into  var[0]=result1, ...  var[N-1]=resultN, var[N]=NULL

internally:  var[0]= ((char*) &var[N+1]),    var[1]= ((char*) &var[N+1])+(strlen(var[0])+1)

so the memory to allocate is:    N+1*sizeof(char*) + Sum(strlen(resultn)+1).

And if you have any porting doubts about this proposal please tell me. I don't
have any, it's pure C. You'd have to cast your malloc result to char** anyway and
adding some extra space after the array doesn't hurt.

afterwards free(var) is all you need !

> > > exec sql type str is varchar[8];
> > > str *var=NULL;
> >
> > Shudder. There should be a decent syntax to specify this. If anyone ever
> > needs this functionality!!!
>
> You're free to add this. It's not trivial though.

Not yet.

> > Propose _if_ ever needed:
> > What about using a different varchar type e.g. [struct varchar_vca  { int
> > len; char *arr; }] which corresponds to  varchar   or   varchar[]  in the
> > parser. But the user should remember to free it afterwards. Sadly we can't
> > default to C++ (destructors).
>
> How should that work? The use only defines a variable as varchar. The rest
> is doen by ecpg. In fact I would prefer if the user does not have to use
> var.arr to access the string but we could find a way to make this completely
> transparent.

I'll take a loom into the standard. Though I don't know when I'll find the time.

> > At first embedded SQL was meant to be portable. And we successfully ported
> > our LNX and Adabas D programs to Postgres. And we share applications between
> > Oracle and Postgres databases.
> > ...

That was just self pitying. And sarcasm about unimplemented or simply never
in-documentation-mentioned standards. [sidekick to big commercial database]

> Don't get me wrong. I did not mean portable to other DBMSs but portable to
> other OSs and C compiler. If it works on all platforms PostgreSQL works on,
> I'm satisfied.

Me too. But then I could have chosen PQ++ from the beginning.
     Christof

PS: Does anybody have interest in SQLDA?

Christof Petig wrote:

> > > > exec sql type str is varchar[8];
> > > > str *var=NULL;
> > >
> > > Shudder. There should be a decent syntax to specify this. If anyone ever
> > > needs this functionality!!!

I found the actual way to get this :

- it is possible to create a pointer to a varchar, simply use the
braindead (to me) syntax:
   varchar type[][maxlen]=0;      or   varchar type[0][maxlen]=0;

- ecpg and the C compiler do not agree on the meaning of
  char *foo[80];
 for the compiler it is an array of 80 char*s for ecpg it's obviously equivalent to char foo[80][0]; (simply try it)
whichis pretty useless for an output variable but might be ok for an input variable if the offset (sizeof char) would
beright (sizeof char*).
 

- the interesting syntax
   char (*foo)[80];   (pointer to an array of char[80])
 is not possible in ecpg, but seems to be the one supported by the lib.


To sum it up: variable declaration and variable initialization are some of the
weakest points in ecpg.

Christof (implementing char **)

PS: casts do not work when initializing variables ( e.g.    int VAR = (int) 1.5; ->
syntax error)

On Tue, Oct 30, 2001 at 10:46:03AM +0100, Christof Petig wrote:
> - it is possible to create a pointer to a varchar, simply use the
> braindead (to me) syntax:
> 
>     varchar type[][maxlen]=0;

Ah yes, the memory comes back. :-9

> - ecpg and the C compiler do not agree on the meaning of

Yup, one more bug to fix.

> PS: casts do not work when initializing variables ( e.g.    int VAR = (int) 1.5; ->
> syntax error)

And another one.

Michael
-- 
Michael Meskes
Michael@Fam-Meskes.De
Go SF 49ers! Go Rhein Fire!
Use Debian GNU/Linux! Use PostgreSQL!

Michael Meskes wrote:

> On Tue, Oct 30, 2001 at 10:46:03AM +0100, Christof Petig wrote:
> > - it is possible to create a pointer to a varchar, simply use the
> > braindead (to me) syntax:
> >
> >     varchar type[][maxlen]=0;
>
> Ah yes, the memory comes back. :-9
>
> > - ecpg and the C compiler do not agree on the meaning of
>
> Yup, one more bug to fix.

I don't care much about these two bugs once the char** patch went in. I might take a
second look at varchar pointers for binary strings. But I'll ask the standard first.

Christof