Andrew Dunstan <andrew@dunslane.net> writes:
> I'd be surprised if using a flex lexer instead made a huge speed
> difference, but maybe I'm wrong.
No, I agree --- it's unlikely to make very much difference in the real
world. Maybe on huge query strings you could notice the difference.
> I'm more concerned that it will be
> difficult to write and maintain and keep in sync with the backend's
> lexical structure - but those are just gut feelings, and I could be way
> off base. Right now psql does just enough recognition to enable it to
> work. Making it recognise the whole sql lexical structure instead
> strikes me as being somewhat redundant - that's what the backend does.
Actually, I thought the way to handle it would be to duplicate the
backend lexer as nearly as possible. Most of the productions would have
empty bodies probably, but doing it that way would guarantee that in
fact psql and the backend would lex a string the same way, which is
exactly the problem we are facing here. You'd fall out of the lexer
only upon detecting backslash (unless we want to put backslash command
lexing into the flex code, which might or might not be a good idea),
or upon detecting a ';' at parenthesis depth 0, or upon hitting end of
string. In the last case the lexer state would indicate which prompt
we need to give.
One unsolved issue in my mind is how to deal with multibyte characters
in non-backend-compatible encodings (the ones where some bytes of a
multibyte character can have values < 128 and thus can look like plain
ASCII characters). We'd have to make sure that the flex lexer skips
over such bytes properly. I've thought of a couple of kluges but
nothing I like ...
regards, tom lane
