Matthew Hagerty <matthew@venux.net> writes:
> Could you give me a few hints on where to start? I have never cracked into
> the pg code. Also, would there be problems with the parser/optimizer
> handling unlimited size queries? I guess my input size relates more to
> having space for large records, etc. but with a bigger input available
> people could write bigger "killer" queries that may choke the parser. Is
> this a problem/concern?
Right, you're beginning to see the size of the project --- it's not just
a matter of changing the size of the query buffer, but of finding all
the downstream places that assume they will never see a query more
complex than X. About the only bright spot is that those limits can
be cured one at a time; you don't need a Big Bang kind of fix. Still,
the job is not done till they are all fixed.
src/interfaces/libpq is the first thing to look at --- it has a buffer
into which the outgoing query must be stuffed. Right now it allocates
an 8K buffer and refuses to deal with anything that doesn't fit. Better
would be to start with maybe a 1K allocation (which would be plenty
for many many apps) and realloc the buffer whenever a larger query is
presented. The input-side buffer is already realloced bigger at need,
so that part is not such a problem. I think the only other trouble spot
in libpq (at least in its core code) is that the error-message buffer is
also fixed size. Long queries could produce long errors, so that buffer
must also become resizable. Eventually; it needn't be your first fix.
Next do the same sort of thing with src/backend/tcop/postgres.c's query
input buffer.
After that you can start stress-testing the backend's parser to see what
problems exist there. I recall someone mentioned a fixed-size buffer in
the lexer, which is produced by flex, so you'll have to read up on flex
to find out how to control that. I have no idea what other limits might
exist in the parser.
The backend error-message handler has fixed size buffers, IIRC, and
after that you can start worrying about the effects of arbitrarily
complex queries on EXPLAIN, pg_dump, the other client apps, the
other client libraries besides libpq, etc etc yadda yadda.
You'll be past a milestone when you can remove the header-file
#defines for MAX_QUERY_BUFFER and friends, but that's a long way
down the pike.
Good luck. This tape will self-destruct in five seconds...
regards, tom lane
