Thread: ISO TESTS for a Pg lexer+parser

Hi!  I am in the process of writing a PostgreSQL lexer/parser in Perl,
because everything else I've found in this area is too buggy.  I'm
basing this lexer/parser on the lexer and parser encoded respectively
in scan.l and gram.y under src/backend/parser.

I'm looking for a solid collection of test cases to test my
lexer/parser.  (I'm hoping that such a collection has already been
produced to test the PostgreSQL yacc-generated parser that comes with
the PostgreSQL distribution.)

(Alternatively, if a non-flaky, rock-solid, and easily customizable Pg
parser in Perl already exists please let me know.)

TIA!

kj

* Kynn Jones (kynnjo@gmail.com) wrote:
> Hi!  I am in the process of writing a PostgreSQL lexer/parser in Perl,
> because everything else I've found in this area is too buggy.  I'm
> basing this lexer/parser on the lexer and parser encoded respectively
> in scan.l and gram.y under src/backend/parser.
>
> I'm looking for a solid collection of test cases to test my
> lexer/parser.  (I'm hoping that such a collection has already been
> produced to test the PostgreSQL yacc-generated parser that comes with
> the PostgreSQL distribution.)
>
> (Alternatively, if a non-flaky, rock-solid, and easily customizable Pg
> parser in Perl already exists please let me know.)

Two things, one, couldn't you just make the lexer/parser in PG into a
C library and then write bindings for it for use under Perl?  Seems like
that'd be more sane.  Second, PG's got a regression test suite that
would probably work decently for testing your lexer/parser.  You might
have to filter out '\' commands, if there are any, and things like that.

    Enjoy,

        Stephen

This stuff:
/postgresql-8.2.4/src/test/regress/sql
Should do for starters.

> -----Original Message-----
> From: pgsql-general-owner@postgresql.org [mailto:pgsql-general-
> owner@postgresql.org] On Behalf Of Kynn Jones
> Sent: Friday, July 06, 2007 12:15 PM
> To: pgsql-general@postgresql.org
> Subject: [GENERAL] ISO TESTS for a Pg lexer+parser
>
> Hi!  I am in the process of writing a PostgreSQL lexer/parser in Perl,
> because everything else I've found in this area is too buggy.  I'm
> basing this lexer/parser on the lexer and parser encoded respectively
> in scan.l and gram.y under src/backend/parser.
>
> I'm looking for a solid collection of test cases to test my
> lexer/parser.  (I'm hoping that such a collection has already been
> produced to test the PostgreSQL yacc-generated parser that comes with
> the PostgreSQL distribution.)
>
> (Alternatively, if a non-flaky, rock-solid, and easily customizable Pg
> parser in Perl already exists please let me know.)
>
> TIA!
>
> kj
>
> ---------------------------(end of
broadcast)---------------------------
> TIP 6: explain analyze is your friend

Stephen Frost wrote:
-- Start of PGP signed section.
> * Kynn Jones (kynnjo@gmail.com) wrote:
> > Hi!  I am in the process of writing a PostgreSQL lexer/parser in Perl,
> > because everything else I've found in this area is too buggy.  I'm
> > basing this lexer/parser on the lexer and parser encoded respectively
> > in scan.l and gram.y under src/backend/parser.
> >
> > I'm looking for a solid collection of test cases to test my
> > lexer/parser.  (I'm hoping that such a collection has already been
> > produced to test the PostgreSQL yacc-generated parser that comes with
> > the PostgreSQL distribution.)
> >
> > (Alternatively, if a non-flaky, rock-solid, and easily customizable Pg
> > parser in Perl already exists please let me know.)
>
> Two things, one, couldn't you just make the lexer/parser in PG into a
> C library and then write bindings for it for use under Perl?  Seems like
> that'd be more sane.  Second, PG's got a regression test suite that
> would probably work decently for testing your lexer/parser.  You might
> have to filter out '\' commands, if there are any, and things like that.

Ditto.  Also consider that the grammer changes in every major PostgreSQL
release.

--
  Bruce Momjian  <bruce@momjian.us>          http://momjian.us
  EnterpriseDB                               http://www.enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

On 7/6/07, Stephen Frost <sfrost@snowman.net> wrote:
> * Kynn Jones (kynnjo@gmail.com) wrote:
> > Hi!  I am in the process of writing a PostgreSQL lexer/parser in Perl,
> > because everything else I've found in this area is too buggy.  I'm
> > basing this lexer/parser on the lexer and parser encoded respectively
> > in scan.l and gram.y under src/backend/parser.
> >
> > I'm looking for a solid collection of test cases to test my
> > lexer/parser.  (I'm hoping that such a collection has already been
> > produced to test the PostgreSQL yacc-generated parser that comes with
> > the PostgreSQL distribution.)

> ...PG's got a regression test suite that
> would probably work decently for testing your lexer/parser.

I found a test suite under

src/test/regress

(thanks to Dann Corbit for the path), but as far as I can tell those
tests exercise only the parser.  Maybe the authors figured that the
lexer would be tested sufficiently, albeit indirectly, through the
tests of the parser?

If I've missed something, I'd be very grateful if someone could tell
me which files in this test suite are devoted to testing the lexer
specifically.

TIA!

kj