Re: Practical impediment to supporting multiple SSL libraries - Mailing list pgsql-hackers

* Martijn van Oosterhout (kleptog@svana.org) wrote:
> Right, I didn't understand that you meant to be doing this
> synchronously, as the data came in. I thought it was just another way
> of retreiving the data already received. But given that a stated reason
> that psqlODBC didn't use the libpq interface was due to the copying of
> all the data, it would be nice if we had something for that. From
> looking at your declaration:
>
> int PQgetTuples(PGresult *res,  // Returns number of tuples populated
>   const int max_ntuples,        // Basically buffer size
>   char *result_set,             // Destination buffer
>   const int *columnOffsets,     // integer array of offsets
>   const int *columnLengths,     // integer array of lengths, for checks
>   const int record_len,         // Length of each structure
>   int *columnNulls,             // 0/1 for is not null / is null
>   int resultFormat);            // Or maybe just binary?
>
> you seem to be suggesting that all the data be stored in one big memory
> block at resultset.

The current block would be stored in one big memory block, yes.
Basically a malloc(BUF_SIZE*sizeof(my_structure));

> What do you do if the data is longer than the given length? What does
> record_len mean (what structures)? Also, you can't specify
> binary/non-binary here, that's done in the query request. libpq doesn't
> handle the data differently depending on binaryness. Also, how can you
> find out the actual length of each value after the call?

hmm, ok, binary/non-binary can be dropped then.  If the data is longer
than the length then you return and let the caller figure out what it
wants to do (realloc, malloc another area, etc).  Record_len is just the
size of each record, so the amount to skip from the start to get to
record #2.  libpq just needs it in getAnotherTuple to calculate the
place to put the next value.  Finding the actual length is a good point,
I should have included (as execParams has) an integer array for this,
which could actually replace columnNulls and have a special indication
when it's a Null value in the column.

> Frankly I'm not seeing much improvement over normal processing. It just
> seems like yet another data-model that won't fit most users. The
> definition of PQgetvalue is merely:
>
> return res->tuples[tup_num][field_num].value;
>
> So we could acheive the same effect by letting people look into
> PQresult before the query is finished. The function you suggest would
> be especially difficult for something like psqlODBC which has no idea
> beforehand how long a value could be.

I don't think it's quite the same effect... :P  I don't think it's
exactly uncommon for people to know their data structure and to have
defined a struct for it, allocate a set of memory and then want to just
loop through the memory in a for() loop based on the structure size.
This has been pretty common in standalone application I've seen and it's
really nice to be able to have a database just dump the results of a
query into such a structure.

> I'm still of the opinion that letting people supply an alternative to
> pqAddTuple would be cleaner. The interface would look like:
>
> typedef struct pgresAttValue
> {
>         int                     len;                    /* length in bytes of the value */
>         char       *value;                      /* actual value, plus terminating zero byte */
> } PGresAttValue;
>
> typedef int (*PQtuplecallback)( PQresult *res, PGresAttValue *fields );
> int PQsettuplecallback( PQresult *res, PQtuplecallback cb );
>
> fields is simply a pointer to an array of nfields such structures.
> Users can do whatever they want with the info, store it in their own
> structure, parse it, throw it away, send it over a network, etc. With
> this callback I could probably implement your function above fairly
> straightforwardly.

Sure you could but you're forced to do more copying around of the data
(copy into the PGresAttValue, copy out of it into your structure array).
If you want something more complex then a callback makes more sense but
I'm of the opinion that we're talking about a 90/10 or 80/20 split here
in terms of dump-into-memory array vs. do-something-more-complicated.
And that opinion isn't *solely* based on Oracle providing a similar
mechanism such that probably quite a few Oracle apps are written exactly
as I suggest (I don't think OCI8 has a callback like you're proposing at
all...).

Just to point out, we could do what you're proposing by letting people
look at PQresult during an async too.. ;)  Except, of course, libpq
would need to allocate/deallocate all the PQresults and have some way of
knowing which have been used by the caller and which havn't.
Thanks,
    Stephen