RE: [HACKERS] Problem with query length - Mailing list pgsql-hackers

Taking this further:

I have discovered why flex is inserting YY_USES_REJECT into the generated .c
file; it's because scan.l uses variable-length trailing context.  This is
mentioned by the flex documentation as a performance degrader of note, as
well as not allowing dynamic token resizing.

So, the question is, can we rewrite scan.l in such a way as to remove the
variable-length trailing context?  I have mailed the flex mailing list to
find out if and how this can be done, so perhaps I'll be able to get rid of
the vltc's and consequently the YY_USES_REJECT.  Then we'll have the dynamic
tokens.  If anybody can help me on this, I'd really appreciate it, because
language parsing is not my strong point.  If anybody can point me to a
reference implementation of SQL, either 92, or 93 (preferably with a
reference lex file) that would probably be a good start.

Natalya,
I suggest that you compile your own source, and make sure that the parser
has its YY_BUF_SIZE set to 64k.  This will at least allow you to use fairly
large tokens, until something else can be done.  The number that you have to
check is in src/backend/parser/Makefile:

scan.c: scan.l       $(LEX) $<       sed -e 's/#define YY_BUF_SIZE .*/#define YY_BUF_SIZE 65536/' \
<lex.yy.c>scan.c       rm -f lex.yy.c
 

Make sure that the 65535 is there, or adjust it to what you need.  It goes
without saying that you need to test this before rolling it into production.

Cheers...

MikeA

>> -----Original Message-----
>> From: Natalya S. Makushina [mailto:mak@rtsoft.msk.ru]
>> Sent: Thursday, August 19, 1999 10:39 AM
>> To: 'Ansley, Michael'; 'pgsql-hackers@postgreSQL.org'
>> Subject: [HACKERS] Problem with query length
>> 
>> 
>>       Hello 
>> 
>> Thank's very much for your help.
>> 
>> I have already installed two copies of PostgreSQL DB. One 
>> was installed from RPM, another one was compiled without 
>> RPM. The copy installed from RPM has problem with query 
>> length, another copy haven't this problem! 
>> 
>> I decide to compile it from sources and try to use. After 
>> compilation all is OK! Query length is 8191 now.
>> 
>>       May be error presents in RPM.
>> 
>> From :        Ansley, Michael 
>> [SMTP:Michael.Ansley@intec.co.za]
>> Date :         18 августа 1999 г. 11:55
>> To :        'Natalya S. Makushina'; 'pgsql-hackers@postgreSQL.org'
>> Subject :        RE: [HACKERS] Problem with query length
>> 
>> Hi, all
>> 
>> I have found out what the problem is, although not (yet) the 
>> solution.  
>> 
>> Executive summary:
>> ------------------
>> The scan.l code is not flexing as intended.  This means 
>> that, for most
>> production installations, the max token size is around 64kB.
>> 
>> Technical summary:
>> ------------------
>> The problem is that scan.l is compiling to scan.c with 
>> YY_USES_REJECT being
>> defined.  When YY_USES_REJECT is defined, the token buffer is NOT
>> expandable, and the parser will fail if expansion is 
>> attempted.  However,
>> YY_USES_REJECT should not be defined, and I'm trying to work 
>> out why it is.
>> I have posted to the flex mailing list, and expect a reply 
>> within the next
>> day or so.
>> 
>> The bottom line:
>> ------------------
>> The token limit seems to be effectively the size of 
>> YY_BUF_SIZE in scan.l,
>> until I submit a patch which should make it unlimited.  
>> 
>> 
>> MikeA
>> 
>> >> -----Original Message-----
>> >> From: Natalya S. Makushina [mailto:mak@rtsoft.msk.ru]
>> >> Sent: Tuesday, August 17, 1999 3:15 PM
>> >> To: 'pgsql-hackers@postgreSQL.org'
>> >> Subject: [HACKERS] Problem with query length
>> >> 
>> >> 
>> >> -------------------------------------------------------------
>> >> ----------------------------------------------------------------
>> >> I have posted this mail to psql-general.  But i didn't get 
>> >> any answer yet.
>> >> -------------------------------------------------------------
>> >> ----------------------------------------------------------------
>> >> 
>> >> When i had tried to insert into text field text (length 
>> >> about 4000 chars), the backend have crashed with status 139. 
>> >> This error is happened when the query length ( SQL query) is 
>> >> more than 4095 chars. I am using PostgreSQL 6.4.2 on Linux.
>> >> 
>> >> My questions are:
>> >> 1. Is there problem with text field or with length of SQL query?
>> >> 2. Would postgresql have any limits for SQL query length?
>> >> I checked the archives but only found references to the 8K 
>> >> limit. Any help would be greatly appreciated.
>> >> Thanks for help
>> >>                 Natalya Makushina 
>> >>                 mak@rtsoft.msk.ru
>> >> 
>> >> 
>>