Thread: nonblocking PQsendQuery will fail for querysize > 8k

nonblocking PQsendQuery will fail for querysize > 8k

From
Andreas Pflug
Date:
Repost from pgadmin-hackers
This happens with 7.3.2 windows libpq.

Regards,

Andreas

> -----Original Message-----
> From: Andreas Pflug [mailto:Andreas.Pflug@web.de] 
> Sent: 31 March 2003 17:57
> To: pgadmin-hackers@postgresql.org
> Subject: Re: [pgadmin-hackers] Maximum PQsendQuery size?
> 
> 
> Andreas Pflug wrote:
> 
> > I'm having trouble with PQsendQuery if sending medium sized 
> queries. 
> > Everything's fine with queries up to 8190 bytes (0x1fffe), but one 
> > byte more will not work. In this case, PQsendQuery and 
> PQconsumeInput 
> > will return ok, but PQisBusy stays busy. How can I get around this?
> >
> > Regards,
> >
> > Andreas
> >
> >
> I already found a solution in pgsql/src/interfaces/libpq/fe-misc.c.
> 
> conn->outBuffer initially has 8k size. For a successful 8190 byte
> execute, this contains a 'Q', the query and the terminating 
> zero byte. 
> If pqPutBytes() finds avail to be too small, it will first try to 
> pqSendSome(), but this will not be good if the connection is 
> nonblocking.
> 
> As a fix, the outbuffer is realloced to be big enough, before 
> processing 
> continues:
> 
>         avail = Max(conn->outBufSize - conn->outCount, 0);  
> >>>>>
>         if (nbytes > avail && pqIsnonblocking(conn))
>         {
>             char *newbuf;
>             conn->outBufSize = conn->outBufSize  - avail + nbytes;
>             newbuf = realloc(conn->outBuffer, conn->outBufSize);
>             conn->outBuffer = newbuf;
>             avail = conn->outBufSize - conn->outCount;
>         }
> <<<<<
>         remaining = Min(avail, nbytes);
> 
> 



Re: nonblocking PQsendQuery will fail for querysize > 8k

From
Tom Lane
Date:
Andreas Pflug <Andreas.Pflug@web.de> writes:
> I'm having trouble with PQsendQuery if sending medium sized 
> queries. Everything's fine with queries up to 8190 bytes (0x1fffe), but one 
> byte more will not work.

libpq's nonblock mode is incredibly fragile, not to say completely
broken.  See discussion from, eg, Jan 2000:
http://archives.postgresql.org/pgsql-hackers/2000-01/msg00964.php
http://archives.postgresql.org/pgsql-hackers/2000-01/msg01048.php
and nearby comments.

AFAIK most of those complaints never did get addressed; so the nonblock
mode works under light load but falls over as soon as it's stressed at
all.  I'd like to see it rewritten properly, but have not had time
myself.

Making pqPutBytes willing to realloc the output buffer bigger would
definitely help, but it's not the only thing needed to make nonblock
mode robust, per comments in above messages.

BTW, reallocing just big enough to handle the current input is not the
way to go; that will result in lots of malloc traffic if the caller adds
a few bytes at a time.  I'd think about doubling the buffer size until
it's big enough.  Your proposed code is also nonrobust about running out
of memory.
        regards, tom lane