Thread: psql, remove include of psqlscan.c

Hi,

This patch (psql_remove_include.patch) eliminates
the #include of psqlscan.c at the bottom of mainloop.c.

The attached patch uses the %top{} flex feature
introduced in flex 2.5.30 released 2003-4-1.
(See the NEWS file for flex.)

The good news is that config/programs.m4
requires flex >= 2.5.31.  The bad news
is that RHEL 5 (released 2007-03-14
with a 13 year (10+3 years) support lifecycle)
has a flex (2.5.4a) that is too old for this patch. :-(
(At least this is what the changelog in flex
from the RHEL 5 srpm repo tells me.)
I don't know what this means.

The flex docs on my box, fwiw, don't mention %top{}
in the section describing Posix (in)compatibility.

The patch is against git head.  All the tests
pass, fwiw.  Because this depends on the toolchain
it wouldn't hurt to build on other architectures/
operating systems.

I'm thinking of exposing enough of the psql parser,
moving it to libpq, that any client-side app can
do what libpq does; given a bunch of sql
separated by semi-colons get the results
of all the statements.  This should also allow
the "statement separation" to be done on the client
side in libpq.  Although I don't imagine that this
will have a performance impact on the server side
it sounds like a first step toward pushing more of
the parsing onto the client.

Regards,

Karl <kop@meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

P.S.  Sent this patch to the list at Sep 27 00:43:28 Central time
and it never showed up so I'm sending again with a different
subject.  Info on the previous message:

Subject:    Remove #include psqlscan.c from psql
message-id=<1348724599.28442.0@mofo>

"Karl O. Pinc" <kop@meme.com> writes:
> This patch (psql_remove_include.patch) eliminates 
> the #include of psqlscan.c at the bottom of mainloop.c.

I don't really see that this is enough of an improvement to justify
depending on a non-portable flex feature.

> I'm thinking of exposing enough of the psql parser,
> moving it to libpq, that any client-side app can
> do what libpq does; given a bunch of sql
> separated by semi-colons get the results
> of all the statements.  This should also allow
> the "statement separation" to be done on the client
> side in libpq.  Although I don't imagine that this
> will have a performance impact on the server side
> it sounds like a first step toward pushing more of
> the parsing onto the client.

Quite honestly, I don't like that idea at all either.  It's bad enough
that psql has to know so much low-level SQL syntax.  If we start
encouraging other programs to know that, we're going to be locked into
never again making any lexical-level changes.  Why exactly is "pushing
parsing onto the client" a good idea?
        regards, tom lane

On 09/27/2012 10:07:48 AM, Tom Lane wrote:
> "Karl O. Pinc" <kop@meme.com> writes:
> > This patch (psql_remove_include.patch) eliminates
> > the #include of psqlscan.c at the bottom of mainloop.c.
>
> I don't really see that this is enough of an improvement to justify
> depending on a non-portable flex feature.

Ok.  (I had no idea it was non-portable.  The flex
docs don't mention this.)

>
> > I'm thinking of exposing enough of the psql parser,
> > moving it to libpq, that any client-side app can
> > do what libpq does; given a bunch of sql
> > separated by semi-colons get the results
> > of all the statements.  This should also allow
> > the "statement separation" to be done on the client
> > side in libpq.  Although I don't imagine that this
> > will have a performance impact on the server side
> > it sounds like a first step toward pushing more of
> > the parsing onto the client.
>
> Quite honestly, I don't like that idea at all either.  It's bad
> enough
> that psql has to know so much low-level SQL syntax.  If we start
> encouraging other programs to know that, we're going to be locked
> into
> never again making any lexical-level changes.  Why exactly is
> "pushing
> parsing onto the client" a good idea?

There's really 2 ideas.  I don't care about this one:
pushing parsing/anything onto the client is a good idea because
clients scale horizontally and so performance is improved.

What I'm thinking of in libpq is the ability to give it big string
with many sql statements and have it hand back each statement
so the client can then submit it to the server for execution.
What I really _want_ is to be able get a bit string of many
sql statements from the user and return the results, statuses,
etc. of executing each statement.  Just what psql does when,
say, fed a file from stdin.

The reason I want this is because I don't want to have to
rewrite the sql parser in PHP for inclusion in phpPgAdmin.
(I did this once, and it was such a big ugly patch
it never got around to getting into the mainline phpPgAdmin.)
phpPgAdmin (pgAdmin/ front-end of your choice)
should be able process a string of sql statements in the
same way that psql does, but currently the only way to
do this is to rewrite the parser in each application.
Much better to have libpq provide the functionality,
although I suppose it's possible to push this into the
server.


Karl <kop@meme.com>
Free Software:  "You don't pay back, you pay forward."                -- Robert A. Heinlein

Excerpts from Karl O. Pinc's message of jue sep 27 12:29:53 -0300 2012:

> The reason I want this is because I don't want to have to
> rewrite the sql parser in PHP for inclusion in phpPgAdmin.
> (I did this once, and it was such a big ugly patch
> it never got around to getting into the mainline phpPgAdmin.)
> phpPgAdmin (pgAdmin/ front-end of your choice)
> should be able process a string of sql statements in the
> same way that psql does, but currently the only way to
> do this is to rewrite the parser in each application.
> Much better to have libpq provide the functionality,
> although I suppose it's possible to push this into the
> server.

This seems a worthy goal to me.

But I think I see what Tom objection to it is: if we "export" this
capability to libpq applications, then we set it in stone to a certain
extent: exactly how things are split would become part of the API, so to
speak.  Upgrading to a newer libpq could break application code that
worked with the previous release's by splitting things differently.

I don't currently have an opinion on whether this is a bad thing or not.
Who wants to argue for/against?

--
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 09/27/12 8:41 AM, Alvaro Herrera wrote:
> But I think I see what Tom objection to it is: if we "export" this
> capability to libpq applications, then we set it in stone to a certain
> extent: exactly how things are split would become part of the API, so to
> speak.  Upgrading to a newer libpq could break application code that
> worked with the previous release's by splitting things differently.
>
> I don't currently have an opinion on whether this is a bad thing or not.
> Who wants to argue for/against?

I wonder if it shouldn't be in a separate 'helper' library, as it has no 
direct ties to any libpq internals.



-- 
john r pierce                            N 37, W 122
santa cruz ca                         mid-left coast

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> But I think I see what Tom objection to it is: if we "export" this
> capability to libpq applications, then we set it in stone to a certain
> extent: exactly how things are split would become part of the API, so to
> speak.  Upgrading to a newer libpq could break application code that
> worked with the previous release's by splitting things differently.

That's not exactly what the problem is: an upgrade could only break
*existing* application code if we had made a non-backward-compatible
change in SQL lexical rules, which hopefully we'd never do.

Rather, the problem is that the server might know about some newer
lexical feature, and so might the application, but if libpq is behind
the times then it's broken.  You can see an instance of this right now
over in the pgsql-jdbc list, where they're arguing about fixing the
JDBC driver to understand dollar quoting.  Application authors not
unreasonably think it should have heard of that by now.

The JDBC example is sort of an argument in favor of Karl's idea,
in that if client-side code is going to know this anyway then it would
be better if it were at least localized in one place.  But of course
JDBC is never going to depend on libpq (wrong language) so moving this
code into libpq isn't actually going to help them.

A larger point is that I don't believe this is actually going to help
anybody, because of mismatch of requirements not only implementation
language.  JDBC couldn't use a libpq lexer implementation even without
the language issue, because the context in which they're arguing about
this is finding and replacing JDBC-spec escape sequences, which libpq is
not going to know about.  I imagine PHP has got the same problem only
different.  Conversely, psql's lexer has a lot of psql-specific design
decisions, such as the need to handle backslash commands and include
files, that I don't think would belong in a libpq implementation.
        regards, tom lane

On 09/27/2012 11:02:42 AM, Tom Lane wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:

> A larger point is that I don't believe this is actually going to help
> anybody, because of mismatch of requirements not only implementation
> language.  JDBC couldn't use a libpq lexer implementation even
> without
> the language issue, because the context in which they're arguing
> about
> this is finding and replacing JDBC-spec escape sequences, which libpq
> is
> not going to know about.  I imagine PHP has got the same problem only
> different.  Conversely, psql's lexer has a lot of psql-specific
> design
> decisions, such as the need to handle backslash commands and include
> files, that I don't think would belong in a libpq implementation.

Well no, I'm not at all interested in escape sequences.
I want to take sql directly from the user and execute it.
Right now I can take only one statement at a time.
And this is true of any human-facing application.

I'm not looking for a general purpose solution, although
it did occur to me that the psql variable substitution
mechanism could be exposed.

But what I really want is not "an exposed parser".  What
I really want is to be able to get results from all the
statements passed to PQexec (et-al), not just the
last statement.  This could be done without exposing
the parser, but it does mean having a parser in libpq.
Since psql links to libpq anyway my plan was to
move the parser entirely out of psql into libpq and
have an undocumented internal interface so that
psql can do the escaping/variable substitution stuff.

Regards,

Karl <kop@meme.com>
Free Software:  "You don't pay back, you pay forward."                -- Robert A. Heinlein

On 09/27/2012 11:02:42 AM, Tom Lane wrote:

> Rather, the problem is that the server might know about some newer
> lexical feature, and so might the application, but if libpq is behind
> the times then it's broken.

If the application knows about the newer feature and wants
to use it, is it unreasonable to ask the application
be linked against a newer libpq?


Karl <kop@meme.com>
Free Software:  "You don't pay back, you pay forward."                -- Robert A. Heinlein

"Karl O. Pinc" <kop@meme.com> writes:
> What I'm thinking of in libpq is the ability to give it big string
> with many sql statements and have it hand back each statement
> so the client can then submit it to the server for execution.
> What I really _want_ is to be able get a bit string of many
> sql statements from the user and return the results, statuses,
> etc. of executing each statement.  Just what psql does when,
> say, fed a file from stdin.

Just as a note --- I believe you can get that result today with
PQsendQuery followed by a PQgetResult loop.  There's no need to
provide an API that splits the string first.
        regards, tom lane

"Karl O. Pinc" <kop@meme.com> writes:
> On 09/27/2012 11:02:42 AM, Tom Lane wrote:
>> Rather, the problem is that the server might know about some newer
>> lexical feature, and so might the application, but if libpq is behind
>> the times then it's broken. 

> If the application knows about the newer feature and wants
> to use it, is it unreasonable to ask the application
> be linked against a newer libpq?

Well, see the JDBC example: there is no newer driver with the desired
feature for applications to link against.  Even if all client-side code
with such knowledge stays perfectly in sync as far as upstream sources
are concerned, there are any number of reasons why particular
installations might have copies of varying vintages.  It's not really
a situation that I want to get into more than necessary.
        regards, tom lane

On 09/27/2012 02:28:49 PM, Tom Lane wrote:
> "Karl O. Pinc" <kop@meme.com> writes:

> > What I really _want_ is to be able get a bit string of many
> > sql statements from the user and return the results, statuses,
> > etc. of executing each statement.  Just what psql does when,
> > say, fed a file from stdin.
>
> Just as a note --- I believe you can get that result today with
> PQsendQuery followed by a PQgetResult loop.  There's no need to
> provide an API that splits the string first.

Ah.  *sigh*  I should pay better attention.  Sorry for the
distraction.


Karl <kop@meme.com>
Free Software:  "You don't pay back, you pay forward."                -- Robert A. Heinlein