Thread: Maximum query string length

Massimo wrote:
>> A better way would be to allocate and grow query buffers dynamically
while
>> reading the query but you will anyway have troubles with lex and yacc
which
>> use statically allocated buffers whose size is hardwired in the program.
>> This is why I had to make those ugly changes in the Makefiles.

Well, if the query length is limited by yacc, bison, lex, or any other tools
that we use, is it worthwhile trying to make it dynamic?  If I can still
only get a 64k query into the backend, what use is there in creating a 800k
query in psql?

Thought and/or bright ideas or welcomed...


MikeA

Ansley, Michael wrote:
> 
> Massimo wrote:
> >> A better way would be to allocate and grow query buffers dynamically
> while
> >> reading the query but you will anyway have troubles with lex and yacc
> which
> >> use statically allocated buffers whose size is hardwired in the program.
> >> This is why I had to make those ugly changes in the Makefiles.
> 
> Well, if the query length is limited by yacc, bison, lex, or any other tools
> that we use, is it worthwhile trying to make it dynamic?  If I can still
> only get a 64k query into the backend, what use is there in creating a 800k
> query in psql?
> 
> Thought and/or bright ideas or welcomed...
> 
> MikeA

That shouldn't be hard to fix. Though it might be hard to fix with good
performance. The default input routines in lex fill its internal buffer
with
a 'line' of text. But all those routines can be overridden. So you could
make lex read directly from the dynamic buffer. Look at yyinput (or is
it yy_input - can't remember).

If you do that, I think the only limitation will be on the size of a
single token. I don't know how postgresql handles quoted text and
comments,
but that would be where the problem may arise.

-- 

Mark Hollomon
mhh@nortelnetworks.com
ESN 451-9008 (302)454-9008

"Ansley, Michael" <Michael.Ansley@intec.co.za> writes:
> Massimo wrote:
>>> you will anyway have troubles with lex and yacc

> Well, if the query length is limited by yacc, bison, lex, or any other tools
> that we use, is it worthwhile trying to make it dynamic?

Yes, I think so.  We have not yet tried hard to persuade those tools to
cooperate, but I find it hard to believe that they cannot be handed a
source string in an externally supplied buffer.  At worst, we might find
that we can only promise > 64K query length when using a bison-generated
parser (since the parser innards tend to vary a lot across vendor
yaccs).

lex may or may not be worth worrying about --- the buffer size limit it
would impose would be for a single token, I believe, not for the whole
query.
        regards, tom lane