Thread: libpq: how to get a sequence of partial PGresult-s

Hi,

I use PostgreSQL as a database for my gui app.
Some queries can take alot of time (e.g. app's search function), and I'd like 
to show recieved data as soon as it arrives. I've played with non-blocking 
mode and PQconsumeInput and as far as I can see, backend sends data (almost?) 
as soon as it's found. The libpq library (protocol v3) reads incoming data 
(DataRow messages) but returns PGresult only when the CommandComplete message 
is recieved. Is there any way to get/process this partial PGresult in my 
app ? I haven't found any API function for this, so I thought about an 
additional function for the libpq's API -

PGresult* PQgetNextResult ( PGconn* conn );

It'd make a tupleless copy of conn->result (PGresult), assign it to conn-
>result and return the old conn->result object, i.e. something like this :

PGresult* PQgetNextResult ( PGconn* conn ) {if ( !conn->result )    return NULL;PGresult* res =
conn->result;conn->result= pgMakeTuplelessCopy ( res );return res;
 
}

It's ok to return an empty PGresult (PQntuples(res) == 0);
NULL return value would still indicate the end of the query.
Libpq would still be able to recieve/process remaining DataRow results.
This would be a better counterpart to mysql's mysql_fetch_row(...) api 
function.

Does this makes sence ?

Btw I know about sql cursors but I'd have to repeat "fetch next from mycursor" 
for all resulting tuples, which is a big processing/network overhead.

-- 
Best regards,
Igor Shevchenko

Igor Shevchenko <igor@carcass.ath.cx> writes:
> Is there any way to get/process this partial PGresult in my 
> app ? I haven't found any API function for this, so I thought about an 
> additional function for the libpq's API -
> PGresult* PQgetNextResult ( PGconn* conn );

I think this is a really bad idea, because it destroys libpq's basic
concept that you get either success or failure for a command.  Peeking
at a partially-arrived result might give you back some rows only to get
an error later (since the backend may fail partway through a query).

I'm not sure that the partially constructed result is valid anyway;
the last row may be in a funny state.  And what happens if the
application does a PQclear() on it??
        regards, tom lane

On Wednesday 24 September 2003 01:27, you wrote:
> Igor Shevchenko <igor@carcass.ath.cx> writes:
> > Is there any way to get/process this partial PGresult in my
> > app ? I haven't found any API function for this, so I thought about an
> > additional function for the libpq's API -
> > PGresult* PQgetNextResult ( PGconn* conn );
>
> I think this is a really bad idea, because it destroys libpq's basic
> concept that you get either success or failure for a command.  Peeking
> at a partially-arrived result might give you back some rows only to get
> an error later (since the backend may fail partway through a query).

Unfortunately there's no way to archieve both streaming an errorless 
PQgetResult, but in case of error, the application will recieve an error 
message later anyway. So, in this context, it can only be application's 
resposibility to rollback changes done by data that is already processed. 
Btw, the same can happen with cursors approach.

> I'm not sure that the partially constructed result is valid anyway;
> the last row may be in a funny state.  And what happens if the
> application does a PQclear() on it??

The application would recieve a deep copy of PGresult, so PQclear() wouldn't 
affect current PGconn->result.

Btw, I was thinking about PQgetNextResult only for the async/non-blocking 
context.

-- 
Best regards,
Igor Shevchenko

As far as I understand, the incomplete PGresult is completed when 
CommandComplete message is recieved.
Basically it only reads/sets conn->result->cmdStatus.
PQreadNextResult would have to set a fake good result (in this usage scenario, 
this is app's responsibility to rollback in case of error even if it appears 
later).

Rows are added only via pqAddTuple(...), and it's called only on valid tuples 
(complete DataRow message is recieved and converted to a tuple 
(PGresAttValue)), so it seems that the last row is always ok.

On Wednesday 24 September 2003 02:00, Igor Shevchenko wrote:
> On Wednesday 24 September 2003 01:27, you wrote:
> > Igor Shevchenko <igor@carcass.ath.cx> writes:
> > > Is there any way to get/process this partial PGresult in my
> > > app ? I haven't found any API function for this, so I thought about an
> > > additional function for the libpq's API -
> > > PGresult* PQgetNextResult ( PGconn* conn );
> >
> > I think this is a really bad idea, because it destroys libpq's basic
> > concept that you get either success or failure for a command.  Peeking
> > at a partially-arrived result might give you back some rows only to get
> > an error later (since the backend may fail partway through a query).
>
> Unfortunately there's no way to archieve both streaming an errorless
> PQgetResult, but in case of error, the application will recieve an error
> message later anyway. So, in this context, it can only be application's
> resposibility to rollback changes done by data that is already processed.
> Btw, the same can happen with cursors approach.
>
> > I'm not sure that the partially constructed result is valid anyway;
> > the last row may be in a funny state.  And what happens if the
> > application does a PQclear() on it??
>
> The application would recieve a deep copy of PGresult, so PQclear()
> wouldn't affect current PGconn->result.
>
> Btw, I was thinking about PQgetNextResult only for the async/non-blocking
> context.

-- 
Best regards,
Igor Shevchenko