Thread: Upgrading our minimum required flex version for 8.5

I'd like to return to the project I suggested here:
http://archives.postgresql.org/message-id/18653.1239741426@sss.pgh.pa.us
of getting rid of plpgsql's private lexer and having it use the core
lexer instead.  This will require making the core lexer re-entrant,
which is not possible with our oldest supported version of flex
(2.5.4a).  A look at the flex change history indicates that %reentrant
was added in 2.5.6 but nasty bugs were being fixed up to 2.5.30;
so the new minimum supported version would probably be 2.5.33
(2.5.31 was kinda broken for other reasons, and there was no 2.5.32).

Since 2.5.33 is now over three years old, this does not seem like an
onerous requirement, but I thought I'd better ask if anyone has an
objection?
        regards, tom lane

Tom Lane wrote:
> I'd like to return to the project I suggested here:
> http://archives.postgresql.org/message-id/18653.1239741426@sss.pgh.pa.us
> of getting rid of plpgsql's private lexer and having it use the core
> lexer instead.  This will require making the core lexer re-entrant,
> which is not possible with our oldest supported version of flex
> (2.5.4a).  A look at the flex change history indicates that %reentrant
> was added in 2.5.6 but nasty bugs were being fixed up to 2.5.30;
> so the new minimum supported version would probably be 2.5.33
> (2.5.31 was kinda broken for other reasons, and there was no 2.5.32).
>
> Since 2.5.33 is now over three years old, this does not seem like an
> onerous requirement, but I thought I'd better ask if anyone has an
> objection?
>
>             
>   



I think it would need to be benchmarked. My faint recollection is that 
the re-entrant lexers are slower.

cheers

andrew

Tom Lane <tgl@sss.pgh.pa.us> wrote:
> so the new minimum supported version would probably be 2.5.33
> (2.5.31 was kinda broken for other reasons, and there was no
> 2.5.32).
> 
> Since 2.5.33 is now over three years old, this does not seem like an
> onerous requirement, but I thought I'd better ask if anyone has an
> objection?
You'd be causing problems for SuSE Enterprise users, like us:
kgrittn@CCDEV-DB:/etc/init.d> cat /etc/SuSE-release
SUSE Linux Enterprise Server 10 (x86_64)
VERSION = 10
PATCHLEVEL = 2
kgrittn@CCDEV-DB:/etc/init.d> flex --version
flex 2.5.31
Just how broken is the version we're using?
-Kevin

"Kevin Grittner" <Kevin.Grittner@wicourts.gov> wrote: 
> Tom Lane <tgl@sss.pgh.pa.us> wrote:
>  
>> so the new minimum supported version would probably be 2.5.33
We still have five SuSE Enterprise 9 boxes in use as database servers;
while we've got someone tasked with replacing them, I wonder if that
version is so dead that we want to abandon support for it.
kgrittn@INHOUSEAPPS:~> cat /etc/SuSE-release
SUSE LINUX Enterprise Server 9 (i586)
VERSION = 9
PATCHLEVEL = 3
kgrittn@INHOUSEAPPS:~> flex --version
flex version 2.5.4
Of course, one could always download source for a more recent version
and build and install flex that way.  (We do that for PostgreSQL
itself and libpoppler.)  If someone's willing to build PostgreSQL,
perhaps it's reasonable to require them to build flex first?
-Kevin

Andrew Dunstan <andrew@dunslane.net> writes:
> I think it would need to be benchmarked. My faint recollection is that 
> the re-entrant lexers are slower.

The flex documentation states in so many words:
The option `--reentrant' does not affect the performance of the scanner.

Do you feel a need to verify their claim?
        regards, tom lane

Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>   
>> I think it would need to be benchmarked. My faint recollection is that 
>> the re-entrant lexers are slower.
>>     
>
> The flex documentation states in so many words:
>
>     The option `--reentrant' does not affect the performance of the scanner.
>
> Do you feel a need to verify their claim?
>
>
>   

No, I'll take their word for it. I must have been thinking of something 
else.

cheers

andrew

Kevin Grittner wrote:
> Tom Lane <tgl@sss.pgh.pa.us> wrote:
>  
> > so the new minimum supported version would probably be 2.5.33
> > (2.5.31 was kinda broken for other reasons, and there was no
> > 2.5.32).
> > 
> > Since 2.5.33 is now over three years old, this does not seem like an
> > onerous requirement, but I thought I'd better ask if anyone has an
> > objection?
>  
> You'd be causing problems for SuSE Enterprise users, like us:
>  
> kgrittn@CCDEV-DB:/etc/init.d> cat /etc/SuSE-release
> SUSE Linux Enterprise Server 10 (x86_64)
> VERSION = 10
> PATCHLEVEL = 2
> kgrittn@CCDEV-DB:/etc/init.d> flex --version
> flex 2.5.31

Keep in mind that if you build from a tarball, the lexer has already
been run elsewhere and you don't need flex.  Since you do some
experimental playing from CVS I guess you could install flex from source
in your environment only ...

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

"Kevin Grittner" <Kevin.Grittner@wicourts.gov> writes:
> kgrittn@CCDEV-DB:/etc/init.d> flex --version
> flex 2.5.31
> Just how broken is the version we're using?

You might want to take that up with SuSE.  The flex NEWS file just cites
"numerous bug and security fixes" from .31 to .33.

Our CVS logs show half a dozen changes we made some years ago to make
our code work with 2.5.31, but I can't tell from the log entries whether
those were workarounds for bugs in flex or fixes for obsolete coding on
our side.

We could try setting the min version at 2.5.31, though, until we find
some specific reason it doesn't work for us.  2.5.31 is significantly
older (release date 2003-04-01, vs 2006-02-20), so there's definitely
more chance of people having it.
        regards, tom lane

Alvaro Herrera <alvherre@commandprompt.com> writes:
> Kevin Grittner wrote:
>> Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> Since 2.5.33 is now over three years old, this does not seem like an
>>> onerous requirement, but I thought I'd better ask if anyone has an
>>> objection?
>> 
>> You'd be causing problems for SuSE Enterprise users, like us:

> Keep in mind that if you build from a tarball, the lexer has already
> been run elsewhere and you don't need flex.

Right, this will only affect people doing development or otherwise
building from a CVS pull.
        regards, tom lane

Tom Lane wrote:
> Alvaro Herrera <alvherre@commandprompt.com> writes:
>   
>> Kevin Grittner wrote:
>>     
>>> Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>>       
>>>> Since 2.5.33 is now over three years old, this does not seem like an
>>>> onerous requirement, but I thought I'd better ask if anyone has an
>>>> objection?
>>>>         
>>> You'd be causing problems for SuSE Enterprise users, like us:
>>>       
>
>   
>> Keep in mind that if you build from a tarball, the lexer has already
>> been run elsewhere and you don't need flex.
>>     
>
> Right, this will only affect people doing development or otherwise
> building from a CVS pull.
>
>             
>   

Of course, that includes the whole buildfarm. We might need to ask some 
people to upgrade there.

cheers

andrew

Alvaro Herrera <alvherre@commandprompt.com> wrote: 
> Keep in mind that if you build from a tarball, the lexer has already
> been run elsewhere and you don't need flex.
Does that hold for the daily snapshots?  If so, I should be good.  My
workstation is OK on flex version.  (I run kubuntu on my desktop.)
-Kevin

Andrew Dunstan <andrew@dunslane.net> writes:
> Tom Lane wrote:
>> Right, this will only affect people doing development or otherwise
>> building from a CVS pull.

> Of course, that includes the whole buildfarm. We might need to ask some 
> people to upgrade there.

Yes.  What I was thinking of doing was committing a configure change to
reject flex < 2.5.31, and waiting to see how much of the buildfarm goes
red.

One point here is that if buildfarm owners update to more recent flex,
that will also affect their testing of back branches.  I think this
should be okay --- AFAICT from the CVS logs, all of the stuff we did to
be compatible with 2.5.31 was pre-7.4 and so all supported versions
should still work.  But we'd find out for sure ;-)
        regards, tom lane

PS: speaking of buildfarm, I don't see any REL8_4_STABLE entries yet.
Need to ping the owners anyway, apparently.

Kevin Grittner wrote:
> Alvaro Herrera <alvherre@commandprompt.com> wrote: 
>  
> > Keep in mind that if you build from a tarball, the lexer has already
> > been run elsewhere and you don't need flex.
>  
> Does that hold for the daily snapshots?  If so, I should be good.

Yes, the snapshots have the derived files too.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

>> Tom Lane wrote:
>>> Right, this will only affect people doing development or otherwise
>>> building from a CVS pull.

Someone doing a CVS pull already needs a specific recent version of
autoconf anyways. How old is this version of flex compared to the
version of autoconf we require?

-- 
greg
http://mit.edu/~gsstark/resume.pdf

On Thu, Jul 2, 2009 at 11:39 AM, Greg Stark<gsstark@mit.edu> wrote:
>>> Tom Lane wrote:
>>>> Right, this will only affect people doing development or otherwise
>>>> building from a CVS pull.
>
> Someone doing a CVS pull already needs a specific recent version of
> autoconf anyways. How old is this version of flex compared to the
> version of autoconf we require?

Really?  configure is in CVS, so I don't think you need autoconf
unless you want to modify configure.in.  Which people may want to do,
but certainly not for every patch.

...Robert

Robert Haas <robertmhaas@gmail.com> writes:
> On Thu, Jul 2, 2009 at 11:39 AM, Greg Stark<gsstark@mit.edu> wrote:
>> Someone doing a CVS pull already needs a specific recent version of
>> autoconf anyways. How old is this version of flex compared to the
>> version of autoconf we require?

> Really?  configure is in CVS, so I don't think you need autoconf
> unless you want to modify configure.in.

Right.  That's been debated on occasion in the past, but right now
only those who commit changes to configure.in (and related files)
need to have autoconf at all.

But to answer Greg's question: 8.4 and HEAD require autoconf 2.61
which was released 2006-11-17, or rather later than flex 2.5.33.
        regards, tom lane

Alvaro Herrera <alvherre@commandprompt.com> wrote: 
> Yes, the snapshots have the derived files too.
Then I take it back -- the new flex versions would have little or no
impact on me.  Worst case, I might need to download a snapshot to
apply my patch for testing on the "big" machines.  If I understood
what make options I could use on my machine to create derived files
for other machines, even that would go away.
-Kevin

"Kevin Grittner" <Kevin.Grittner@wicourts.gov> writes:
> Then I take it back -- the new flex versions would have little or no
> impact on me.  Worst case, I might need to download a snapshot to
> apply my patch for testing on the "big" machines.  If I understood
> what make options I could use on my machine to create derived files
> for other machines, even that would go away.

Peter or Marc could clue you in a bit better, but I think it's as simple
as saying "make dist" at the top level of a modified source tree.  This
gets you a source tarball the same way the release tarballs are made.
        regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> wrote: 
> Peter or Marc could clue you in a bit better, but I think it's as
> simple as saying "make dist" at the top level of a modified source
> tree.  This gets you a source tarball the same way the release
> tarballs are made.
I seem to be able to "sneak up on it from behind" by doing a regular
make and then "make distclean" and copying the results.  Perhaps
someone knows off-hand what I'm missing that prevents "make dist" from
working.  The attempt ends with:
make[3]: Entering directory
`/home/kgrittn/pg/pgsql/postgresql-8.5devel/contrib/xml2'
make[3]: Nothing to be done for `distprep'.
make[3]: Leaving directory
`/home/kgrittn/pg/pgsql/postgresql-8.5devel/contrib/xml2'
make[2]: Leaving directory
`/home/kgrittn/pg/pgsql/postgresql-8.5devel/contrib'
make[1]: Leaving directory
`/home/kgrittn/pg/pgsql/postgresql-8.5devel'
make -C postgresql-8.5devel/doc/src/sgml/ HISTORY INSTALL
regress_README
make[1]: Entering directory
`/home/kgrittn/pg/pgsql/postgresql-8.5devel/doc/src/sgml'
"/usr/bin/perl" generate_history.pl "." release.sgml
>tempfile_HISTORY.sgml
jade  -D .  -d stylesheet.dsl -i output-text -t sgml -V nochunks
tempfile_HISTORY.sgml >HISTORY.html
/bin/sh: jade: not found
make[1]: *** [HISTORY.html] Error 127
make[1]: *** Deleting file `HISTORY.html'
make[1]: Leaving directory
`/home/kgrittn/pg/pgsql/postgresql-8.5devel/doc/src/sgml'
make: *** [distdir] Error 2
-Kevin

I wrote:
> Yes.  What I was thinking of doing was committing a configure change to
> reject flex < 2.5.31, and waiting to see how much of the buildfarm goes
> red.

Actually, most of the buildfarm members show which flex version they are
running in the configure output.  A quick look shows that of the 45
members that have reported on HEAD in the past 2 days, 22 are running
2.5.4, which is a lot higher than I was expecting.  Most of these are
the Solaris boxen, which I imagine can be updated fairly painlessly
since there are some of them that are already running something newer.
However I'm a bit worried about the situation for Windows --- does
anyone know whether a newer flex is readily available for Windows?

In any case it seems like it'd be prudent to prod the buildfarm owners
to update their flex *before* we pull the trigger...
        regards, tom lane

"Kevin Grittner" <Kevin.Grittner@wicourts.gov> writes:
> I seem to be able to "sneak up on it from behind" by doing a regular
> make and then "make distclean" and copying the results.  Perhaps
> someone knows off-hand what I'm missing that prevents "make dist" from
> working.  The attempt ends with:

> jade  -D .  -d stylesheet.dsl -i output-text -t sgml -V nochunks
> tempfile_HISTORY.sgml >HISTORY.html
> /bin/sh: jade: not found

Looks like you're missing openjade.  However, that's only used to
produce the HISTORY file which is hardly critical, so copying your
source tree after building is probably a perfectly good answer.
        regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> wrote: 
> "Kevin Grittner" <Kevin.Grittner@wicourts.gov> writes:
>> jade  -D .  -d stylesheet.dsl -i output-text -t sgml -V nochunks
>> tempfile_HISTORY.sgml >HISTORY.html
>> /bin/sh: jade: not found
> 
> Looks like you're missing openjade.  However, that's only used to
> produce the HISTORY file which is hardly critical, so copying your
> source tree after building is probably a perfectly good answer.
Thanks.  I was able to install jade, but it throws pages of errors.
Rather than try to get that working, I'll just count on the other
approach.  I'd want to make it and check it on my machine first,
anyway, so the distclean is no big deal at that point.
-Kevin

On Thursday 02 July 2009 19:46:04 Tom Lane wrote:
> "Kevin Grittner" <Kevin.Grittner@wicourts.gov> writes:
> > Then I take it back -- the new flex versions would have little or no
> > impact on me.  Worst case, I might need to download a snapshot to
> > apply my patch for testing on the "big" machines.  If I understood
> > what make options I could use on my machine to create derived files
> > for other machines, even that would go away.
>
> Peter or Marc could clue you in a bit better, but I think it's as simple
> as saying "make dist" at the top level of a modified source tree.  This
> gets you a source tarball the same way the release tarballs are made.

make distprep is what prepares those files.  make dist depends on make 
distprep and then builds a tarball out of everything.

On Thu, Jul 2, 2009 at 12:13 PM, Tom Lane<tgl@sss.pgh.pa.us> wrote:
> However I'm a bit worried about the situation for Windows --- does
> anyone know whether a newer flex is readily available for Windows?
>

MSYS Suplementary Tools (for mingw) includes flex-2.5.33
http://sourceforge.net/projects/mingw/files/

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

Peter Eisentraut <peter_e@gmx.net> wrote: 
> make distprep is what prepares those files.
Perfect!  Flex files built.  No errors.  Thanks much!
-Kevin

Tom Lane píše v čt 02. 07. 2009 v 11:32 -0400:
> Andrew Dunstan <andrew@dunslane.net> writes:
> > Tom Lane wrote:
> >> Right, this will only affect people doing development or otherwise
> >> building from a CVS pull.
> 
> > Of course, that includes the whole buildfarm. We might need to ask some 
> > people to upgrade there.
> 
> Yes.  What I was thinking of doing was committing a configure change to
> reject flex < 2.5.31, and waiting to see how much of the buildfarm goes
> red.

Solaris 10 has 2.5.4 version. OpenSolaris already has 2.5.33.
    Zdenek

Tom Lane píše v čt 02. 07. 2009 v 13:13 -0400:
> I wrote:
> > Yes.  What I was thinking of doing was committing a configure change to
> > reject flex < 2.5.31, and waiting to see how much of the buildfarm goes
> > red.
> 
> Actually, most of the buildfarm members show which flex version they are
> running in the configure output.  A quick look shows that of the 45
> members that have reported on HEAD in the past 2 days, 22 are running
> 2.5.4, which is a lot higher than I was expecting.  Most of these are
> the Solaris boxen, which I imagine can be updated fairly painlessly
> since there are some of them that are already running something newer.

I hope, need to check/compile, but I don't expect any problems.
Zdenek

Tom Lane píše v čt 02. 07. 2009 v 13:13 -0400:

> 
> Actually, most of the buildfarm members show which flex version they are
> running in the configure output.  A quick look shows that of the 45
> members that have reported on HEAD in the past 2 days, 22 are running
> 2.5.4, which is a lot higher than I was expecting.  Most of these are
> the Solaris boxen, which I imagine can be updated fairly painlessly
> since there are some of them that are already running something newer.

My three S10 are updated. All of them use version 2.5.35 now. S8, S9 is
not yet under my control and they are not migrated yet to new location. 
    Zdenek

Andrew Dunstan <andrew@dunslane.net> writes:
> Tom Lane wrote:
>> Andrew Dunstan <andrew@dunslane.net> writes:
>>> I think it would need to be benchmarked. My faint recollection is that 
>>> the re-entrant lexers are slower.
>> 
>> The flex documentation states in so many words:
>>    The option `--reentrant' does not affect the performance of the scanner.
>> Do you feel a need to verify their claim?

> No, I'll take their word for it. I must have been thinking of something 
> else.

As I got further into this, it turned out that Andrew's instinct was
right: it does need to be benchmarked.  Although the inner loops of the
lexer seem to be the same with or without --reentrant, once you buy into
the whole nine yards of --reentrant, --bison-bridge, and a "pure" bison
parser, you find out that the lexer's API changes: there are more
parameters to yylex() than there used to be.  It's also necessary to
pass around a yyscanner pointer to all the subroutines in scan.l.  (But
on the other hand, this eliminates accesses to global variables, which
are often not that cheap.)  So the "no performance impact" claim isn't
telling the whole truth.

As best I can tell after some casual testing on a couple of machines,
the actual bottom line is that "raw_parser" (ie, the bison and flex
processing) is going to be a couple of percent slower with a reentrant
grammar and lexer, for typical queries involving a lot of short tokens.
Now this disappears into the noise as soon as you include parse analysis
(let alone planning and execution), but it is possible to measure the
slowdown in a test harness that calls raw_parser only.

A possible compromise that I think would avoid most or all of the
slowdown is to make the lexer reentrant but not the grammar (so that
yylval and yylloc remain as global variables instead of being parameters
to yylex).  I haven't actually benchmarked that, though.  It strikes
me as a fairly silly thing to do.  If we're going to go for reentrancy
I think we should fix both components.

I'm willing to live with the small slowdown.  Comments?
        regards, tom lane

2009/7/12 Tom Lane <tgl@sss.pgh.pa.us>:
> Andrew Dunstan <andrew@dunslane.net> writes:
>> Tom Lane wrote:
>>> Andrew Dunstan <andrew@dunslane.net> writes:
>>>> I think it would need to be benchmarked. My faint recollection is that
>>>> the re-entrant lexers are slower.
>>>
>>> The flex documentation states in so many words:
>>>    The option `--reentrant' does not affect the performance of the scanner.
>>> Do you feel a need to verify their claim?
>
>> No, I'll take their word for it. I must have been thinking of something
>> else.
>
> As I got further into this, it turned out that Andrew's instinct was
> right: it does need to be benchmarked.  Although the inner loops of the
> lexer seem to be the same with or without --reentrant, once you buy into
> the whole nine yards of --reentrant, --bison-bridge, and a "pure" bison
> parser, you find out that the lexer's API changes: there are more
> parameters to yylex() than there used to be.  It's also necessary to
> pass around a yyscanner pointer to all the subroutines in scan.l.  (But
> on the other hand, this eliminates accesses to global variables, which
> are often not that cheap.)  So the "no performance impact" claim isn't
> telling the whole truth.
>
> As best I can tell after some casual testing on a couple of machines,
> the actual bottom line is that "raw_parser" (ie, the bison and flex
> processing) is going to be a couple of percent slower with a reentrant
> grammar and lexer, for typical queries involving a lot of short tokens.
> Now this disappears into the noise as soon as you include parse analysis
> (let alone planning and execution), but it is possible to measure the
> slowdown in a test harness that calls raw_parser only.
>
> A possible compromise that I think would avoid most or all of the
> slowdown is to make the lexer reentrant but not the grammar (so that
> yylval and yylloc remain as global variables instead of being parameters
> to yylex).  I haven't actually benchmarked that, though.  It strikes
> me as a fairly silly thing to do.  If we're going to go for reentrancy
> I think we should fix both components.

when we don't use reentrant grammar, then we cannot use main sql parser in SQL?

Pavel

>
> I'm willing to live with the small slowdown.  Comments?
>
>                        regards, tom lane
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

Tom Lane wrote:
> As best I can tell after some casual testing on a couple of machines,
> the actual bottom line is that "raw_parser" (ie, the bison and flex
> processing) is going to be a couple of percent slower with a reentrant
> grammar and lexer, for typical queries involving a lot of short tokens.
> Now this disappears into the noise as soon as you include parse analysis
> (let alone planning and execution), but it is possible to measure the
> slowdown in a test harness that calls raw_parser only.
>
> A possible compromise that I think would avoid most or all of the
> slowdown is to make the lexer reentrant but not the grammar (so that
> yylval and yylloc remain as global variables instead of being parameters
> to yylex).  I haven't actually benchmarked that, though.  It strikes
> me as a fairly silly thing to do.  If we're going to go for reentrancy
> I think we should fix both components.
>
> I'm willing to live with the small slowdown.  Comments?
>
>             
>   


If we're going to have a reentrant lexer, I think we should go the whole 
nine yards. I agree that a couple of percent slowdown on just the lexing 
and parsing will be lost in the noise. So +1 from me.

cheers

andrew

Pavel Stehule <pavel.stehule@gmail.com> writes:
> 2009/7/12 Tom Lane <tgl@sss.pgh.pa.us>:
>> If we're going to go for reentrancy
>> I think we should fix both components.

> when we don't use reentrant grammar, then we cannot use main sql parser in SQL?

It wouldn't be a problem for the immediate application I have in mind,
which is to re-use the core lexer in plpgsql.  But it does seem like
it might be a problem down the road as plpgsql gets smarter.
        regards, tom lane

2009/7/12 Tom Lane <tgl@sss.pgh.pa.us>:
> Pavel Stehule <pavel.stehule@gmail.com> writes:
>> 2009/7/12 Tom Lane <tgl@sss.pgh.pa.us>:
>>> If we're going to go for reentrancy
>>> I think we should fix both components.
>
>> when we don't use reentrant grammar, then we cannot use main sql parser in SQL?
>
> It wouldn't be a problem for the immediate application I have in mind,
> which is to re-use the core lexer in plpgsql.  But it does seem like
> it might be a problem down the road as plpgsql gets smarter.
>

it's bad. I thing so integration main parser into plpgsql should be
the most important feature of plpgsql from trapping exception time. I
have to ask - we need it necessary reetrant grammer? We need
integration only in complilation time - for CREATE FUNCTION statement.
Can be nonreetrant grammer problem (but we have to store some info
from validation time somewhere - maybe in probin column) ?

>                        regards, tom lane
>

Andrew Dunstan <andrew@dunslane.net> writes:
> If we're going to have a reentrant lexer, I think we should go the whole
> nine yards. I agree that a couple of percent slowdown on just the lexing
> and parsing will be lost in the noise. So +1 from me.

I found a couple of places where a few cycles could be shaved.  The
current version of the patch (attached) seems indistinguishable in
speed from 8.4 on my HPPA box, though still a percent or two slower on
my x86_64 box.

This is ready to go except for changing the minimum flex version test
in configure (and associated documentation).  I see that a good-sized
fraction of the buildfarm is still on flex 2.5.4 and will therefore go
red when this goes in.  Should I hold off a bit longer, or is committing
the best way to get their attention?

            regards, tom lane

Tom Lane wrote:
> I see that a good-sized
> fraction of the buildfarm is still on flex 2.5.4 and will therefore go
> red when this goes in.  Should I hold off a bit longer, or is committing
> the best way to get their attention?
>
>   

Probably the latter. I will update my various members. I see that 2.5.4 
is the default on my FBSD install, which is the latest, so for this and 
maybe some other current platforms we'll be imposing a bit of extra 
burden to build, but I guess that's the price of progress.

cheers

andrew

> This is ready to go except for changing the minimum flex version test
> in configure (and associated documentation).  I see that a good-sized
> fraction of the buildfarm is still on flex 2.5.4 and will therefore go
> red when this goes in.  Should I hold off a bit longer, or is committing
> the best way to get their attention?

Oh, I didn't think about the Flex version.  That's going to be a far 
more widespread problem.  OSX 10.5, for example, still ships with 
2.5.33.  I suspect that a *lot* of OSes won't have anything up-to-date.

-- 
Josh Berkus
PostgreSQL Experts Inc.
www.pgexperts.com

Josh Berkus wrote:
>
>> This is ready to go except for changing the minimum flex version test
>> in configure (and associated documentation).  I see that a good-sized
>> fraction of the buildfarm is still on flex 2.5.4 and will therefore go
>> red when this goes in.  Should I hold off a bit longer, or is committing
>> the best way to get their attention?
>
> Oh, I didn't think about the Flex version.  That's going to be a far 
> more widespread problem.  OSX 10.5, for example, still ships with 
> 2.5.33.  I suspect that a *lot* of OSes won't have anything up-to-date.
>

That's the version Tom is proposing to make the minimum.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:
> Josh Berkus wrote:
>> Oh, I didn't think about the Flex version.  That's going to be a far 
>> more widespread problem.  OSX 10.5, for example, still ships with 
>> 2.5.33.  I suspect that a *lot* of OSes won't have anything up-to-date.

> That's the version Tom is proposing to make the minimum.

Yeah, 2.5.33 is fine (in fact it's what I installed on my HPUX box to
replace 2.5.4).  AFAICS OSX 10.5 is fine, but 10.4 will need a newer
flex version.  This is not anything very new to Mac users.  10.4 also
shipped with bison 1.28, which is too old to build Postgres and has been
too old for many years now.  So anyone building from CVS on that
platform has already installed at least one nondefault tool.

(For comparison's sake: flex 2.5.4 was released in 1996.  2.5.31, which
I'm proposing to make the new minimum, was released in 2003.  Bison
1.875, our current minimum for that tool, was released in 2002, and
we made it the minimum required version in 2003.)
        regards, tom lane

Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Now this disappears into the noise as soon as you include parse
> analysis (let alone planning and execution)
No need to go for silly options to avoid a performance issue at that
level.  Time wasted dealing with subsequent silliness would almost
certainly have a higher "lost opportunity" cost regarding things that
matter more.
-Kevin

Andrew Dunstan wrote:
>
>
> Tom Lane wrote:
>> I see that a good-sized
>> fraction of the buildfarm is still on flex 2.5.4 and will therefore go
>> red when this goes in.  Should I hold off a bit longer, or is committing
>> the best way to get their attention?
>>
>>   
>
> Probably the latter. I will update my various members. I see that 
> 2.5.4 is the default on my FBSD install, which is the latest, so for 
> this and maybe some other current platforms we'll be imposing a bit of 
> extra burden to build, but I guess that's the price of progress.
>
>

OK, the fly in this ointment turns out to be MSVC. The latest flex from 
GnuWin32 is 2.5.4a, and building 2.5.35 for Windows is turning out to be 
quite a pain. Luckily, MinGW has a pre-built modified 2.5.33 available, 
and I have installed this (also needed msys-regex), and then I jury 
rigged my MSVC build to use it, so we still have one MSVC working OK in 
the buildfarm. But we can hardly ask people to install MinGW/MSys so 
they can build with MSVC, that's horribly ugly. I'll work on getting 
version 2.5.35 build for Windows in a way that works standalone, and 
push it somewhere (maybe the dev wiki).

cheers

andrew

Andrew Dunstan wrote:
>
> OK, the fly in this ointment turns out to be MSVC. The latest flex 
> from GnuWin32 is 2.5.4a, and building 2.5.35 for Windows is turning 
> out to be quite a pain. Luckily, MinGW has a pre-built modified 2.5.33 
> available, and I have installed this (also needed msys-regex), and 
> then I jury rigged my MSVC build to use it, so we still have one MSVC 
> working OK in the buildfarm. But we can hardly ask people to install 
> MinGW/MSys so they can build with MSVC, that's horribly ugly. I'll 
> work on getting version 2.5.35 build for Windows in a way that works 
> standalone, and push it somewhere (maybe the dev wiki).
>
>

Well, it looks like there's a reason GnuWin32 hasn't advanced beyond 
2.5.4a - after that the flex developers proceeded to make flex use a 
filter chain methodology that requires the use of fork(). Making it run  
on Windows without the  support of Msys or Cygwin would involve some 
significant surgery, I suspect.

Maybe for the time being we need to think about keeping scan.c in CVS. 
It's not like scan.l gets updated all that often.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:
> Well, it looks like there's a reason GnuWin32 hasn't advanced beyond 
> 2.5.4a - after that the flex developers proceeded to make flex use a 
> filter chain methodology that requires the use of fork(). Making it run  
> on Windows without the  support of Msys or Cygwin would involve some 
> significant surgery, I suspect.

Egad, this is a mess :-(.  I noticed in the flex changelog that they'd
switched to using m4 instead of implementing all the text processing
themselves.  I suppose this is a consequence of that.

But I'm not prepared to agree that M$ lameness should restrict us to
using only a 1990s version of flex.  Didn't somebody mention upthread
that there is a Windows port of 2.5.33 available?

> Maybe for the time being we need to think about keeping scan.c in CVS. 
> It's not like scan.l gets updated all that often.

We could if we had to, though it amounts to saying that Windows-based
developers don't get to touch the scanner.
        regards, tom lane

Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>   
>> Well, it looks like there's a reason GnuWin32 hasn't advanced beyond 
>> 2.5.4a - after that the flex developers proceeded to make flex use a 
>> filter chain methodology that requires the use of fork(). Making it run  
>> on Windows without the  support of Msys or Cygwin would involve some 
>> significant surgery, I suspect.
>>     
>
> Egad, this is a mess :-(.  I noticed in the flex changelog that they'd
> switched to using m4 instead of implementing all the text processing
> themselves.  I suppose this is a consequence of that.
>
> But I'm not prepared to agree that M$ lameness should restrict us to
> using only a 1990s version of flex.  Didn't somebody mention upthread
> that there is a Windows port of 2.5.33 available?
>   

Not one that is standalone. It needs some runtime support for Unix-like 
process control, either Cygwin or Msys. So my Cygwin and Mingw/MSys 
buildfarm members are quite happily using flex built for their 
respective environments. I am leveraging the fact that I run all three 
of my Windows based buildfarm members on the same Windows instance to 
let the MSVC member use the MSys flex.
>   
>> Maybe for the time being we need to think about keeping scan.c in CVS. 
>> It's not like scan.l gets updated all that often.
>>     
>
> We could if we had to, though it amounts to saying that Windows-based
> developers don't get to touch the scanner.
>
>
>   

True, but I'm not going to invest a large number of cycles on porting 
this. I'm not very happy about it either. I guess anyone wanting to 
develop on Windows and affect the scanner could install Cygwin or MSys.

cheers

andrew

> -----Original Message-----
> From: pgsql-hackers-owner@postgresql.org [mailto:pgsql-hackers-
> owner@postgresql.org] On Behalf Of Tom Lane
> Sent: Monday, July 13, 2009 7:43 PM
> To: Andrew Dunstan
> Cc: pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] Upgrading our minimum required flex version for
> 8.5
>
> Andrew Dunstan <andrew@dunslane.net> writes:
> > Well, it looks like there's a reason GnuWin32 hasn't advanced beyond
> > 2.5.4a - after that the flex developers proceeded to make flex use a
> > filter chain methodology that requires the use of fork(). Making it
> run
> > on Windows without the  support of Msys or Cygwin would involve some
> > significant surgery, I suspect.
>
> Egad, this is a mess :-(.  I noticed in the flex changelog that they'd
> switched to using m4 instead of implementing all the text processing
> themselves.  I suppose this is a consequence of that.
>
> But I'm not prepared to agree that M$ lameness should restrict us to
> using only a 1990s version of flex.  Didn't somebody mention upthread
> that there is a Windows port of 2.5.33 available?
>
> > Maybe for the time being we need to think about keeping scan.c in
> CVS.
> > It's not like scan.l gets updated all that often.
>
> We could if we had to, though it amounts to saying that Windows-based
> developers don't get to touch the scanner.
>
>             regards, tom lane


Flex 2.5.33 and bison 2.3 are available from mingw for windows.

http://sourceforge.net/projects/mingw/files/

Since mingw programs don't need Cygwin installed, these should probably be OK for most Windows people.

But if really needed, flex 2.5.33 could be ported (m4 is already ported).

I'm also wonderings why flex is a problem, since there is a GNUwin32 (native) port of Bison 2.4.1 and m4,
And Bison uses m4 these days, doesn't it?  Perhaps it wouldn't be so hard to update flex to use the same m4 calling
thatbison uses? 



Chuck McDevitt

Chuck McDevitt wrote:
>
> Flex 2.5.33 and bison 2.3 are available from mingw for windows.
>
> http://sourceforge.net/projects/mingw/files/
>
> Since mingw programs don't need Cygwin installed, these should probably be OK for most Windows people.
>
> But if really needed, flex 2.5.33 could be ported (m4 is already ported).
>
> I'm also wonderings why flex is a problem, since there is a GNUwin32 (native) port of Bison 2.4.1 and m4,
> And Bison uses m4 these days, doesn't it?  Perhaps it wouldn't be so hard to update flex to use the same m4 calling
thatbison uses?
 
>
>
>   

It is part of the MSys utilities, and requires that you have MSys 
installed. MSys is in effect a Cygwin equivalent. The flex from MinGW 
will NOT run standalone. I have tried.

If you think it can be ported to run standalone (i.e. without either 
Cygwin or MSys), have at it. Or take my advice and don't waste time trying.

cheers

andrew

Andrew Dunstan wrote:
>
>
> Chuck McDevitt wrote:
>>
>> Flex 2.5.33 and bison 2.3 are available from mingw for windows.
>>
>> http://sourceforge.net/projects/mingw/files/
>>
>> Since mingw programs don't need Cygwin installed, these should 
>> probably be OK for most Windows people.
>>
>> But if really needed, flex 2.5.33 could be ported (m4 is already 
>> ported).
>>
>> I'm also wonderings why flex is a problem, since there is a GNUwin32 
>> (native) port of Bison 2.4.1 and m4,
>> And Bison uses m4 these days, doesn't it?  Perhaps it wouldn't be so 
>> hard to update flex to use the same m4 calling that bison uses?
>>
>>
>>   
>
> It is part of the MSys utilities, and requires that you have MSys 
> installed. MSys is in effect a Cygwin equivalent. The flex from MinGW 
> will NOT run standalone. I have tried.
>
> If you think it can be ported to run standalone (i.e. without either 
> Cygwin or MSys), have at it. Or take my advice and don't waste time 
> trying.
>
>


What I have done is built a version of flex 2.5.35 with Cygwin, and 
bundled the .exe with the Cygwin DLL (the only other thing it should 
need, for our purposes) in a zip file. It can currently be fetched from 
<http://developer.postgresql.org/~adunstan/windows-flex-2.5.35.zip>. 
That way people won't have to do a full Cygwin or MSys install to use it.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:
> What I have done is built a version of flex 2.5.35 with Cygwin, and 
> bundled the .exe with the Cygwin DLL (the only other thing it should 
> need, for our purposes) in a zip file. It can currently be fetched from 
> <http://developer.postgresql.org/~adunstan/windows-flex-2.5.35.zip>. 
> That way people won't have to do a full Cygwin or MSys install to use it.

Sounds good.  The other idea that was suggested (putting scan.c into
CVS) seems problematic on reflection.  I don't believe that CVS will
guarantee that scan.c has a same-or-newer timestamp as scan.l if both
are pulled from CVS; so you might find make trying to rebuild scan.c
anyway.
        regards, tom lane

On Tue, Jul 14, 2009 at 05:10, Andrew Dunstan<andrew@dunslane.net> wrote:
> Tom Lane wrote:

>>> Maybe for the time being we need to think about keeping scan.c in CVS.
>>> It's not like scan.l gets updated all that often.
>>>
>>
>> We could if we had to, though it amounts to saying that Windows-based
>> developers don't get to touch the scanner.
>>
>>
>>
>
> True, but I'm not going to invest a large number of cycles on porting this.
> I'm not very happy about it either. I guess anyone wanting to develop on
> Windows and affect the scanner could install Cygwin or MSys.

I think requiring that for messing with the scanner is acceptable. As
it is now, requiring that to do *any* development or compiling on
HEAD, is a serious regression.

FWIW, it seems the version that Andrew put up doesn't work in one of
my test environments, and also not in at last one of Dave's. I will
test it in my second test environment later today to be sure.

-- Magnus HaganderSelf: http://www.hagander.net/Work: http://www.redpill-linpro.com/

On Wed, Jul 22, 2009 at 12:42 PM, Magnus Hagander<magnus@hagander.net> wrote:

> FWIW, it seems the version that Andrew put up doesn't work in one of
> my test environments, and also not in at last one of Dave's. I will
> test it in my second test environment later today to be sure.

It doesn't work in any of my bf animals, or build machines :-(

-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

Dave Page wrote:
> On Wed, Jul 22, 2009 at 12:42 PM, Magnus Hagander<magnus@hagander.net> wrote:
>
>   
>> FWIW, it seems the version that Andrew put up doesn't work in one of
>> my test environments, and also not in at last one of Dave's. I will
>> test it in my second test environment later today to be sure.
>>     
>
> It doesn't work in any of my bf animals, or build machines :-(
>   


Darn. I'm working on it.

cheers

andrew

Dave Page <dpage@pgadmin.org> writes:
> It doesn't work in any of my bf animals, or build machines :-(

?? narwhal seems to have gone green.
        regards, tom lane

On Wed, Jul 22, 2009 at 2:16 PM, Tom Lane<tgl@sss.pgh.pa.us> wrote:
> Dave Page <dpage@pgadmin.org> writes:
>> It doesn't work in any of my bf animals, or build machines :-(
>
> ?? narwhal seems to have gone green.

Narwhal is mingw/msys. The misbehaving flex is the one provided by
Andrew for use with VC++ (where the aim is to avoid having mingw/msys
of course).


-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

Tom Lane wrote:
> Dave Page <dpage@pgadmin.org> writes:
>   
>> It doesn't work in any of my bf animals, or build machines :-(
>>     
>
> ?? narwhal seems to have gone green.
>   


Yeah, the problem is with MSVC, Narwal is a Mingw box.

cheers

andrew

I got bit by this tonight as part of testing a patch on CentOS 5, which 
like RHEL 5 still ships flex 2.5.4.  I just wrote a little guide on how to 
grab a source RPM from a Fedora version and install it to work around that 
problem: 
http://notemagnet.blogspot.com/2009/07/upgrading-flex-from-source-rpm-to.html

Kind of annoying, but as special software you have to install on a server 
just to build something from CVS goes it's only a minor inconvenience.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

On Thu, Jul 23, 2009 at 2:57 AM, Greg Smith<gsmith@gregsmith.com> wrote:
> I got bit by this tonight as part of testing a patch on CentOS 5, which like
> RHEL 5 still ships flex 2.5.4.  I just wrote a little guide on how to grab a
> source RPM from a Fedora version and install it to work around that problem:
> http://notemagnet.blogspot.com/2009/07/upgrading-flex-from-source-rpm-to.html
>
> Kind of annoying, but as special software you have to install on a server
> just to build something from CVS goes it's only a minor inconvenience.

*sigh*. So does OS X Tiger. And (unsurprisingly), CentOS 4, which is
what we use for all our Linux build servers.

This is getting quite tiresome :-(

--
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

On Thu, Jul 23, 2009 at 10:52, Dave Page<dpage@pgadmin.org> wrote:
> On Thu, Jul 23, 2009 at 2:57 AM, Greg Smith<gsmith@gregsmith.com> wrote:
>> I got bit by this tonight as part of testing a patch on CentOS 5, which like
>> RHEL 5 still ships flex 2.5.4.  I just wrote a little guide on how to grab a
>> source RPM from a Fedora version and install it to work around that problem:
>> http://notemagnet.blogspot.com/2009/07/upgrading-flex-from-source-rpm-to.html
>>
>> Kind of annoying, but as special software you have to install on a server
>> just to build something from CVS goes it's only a minor inconvenience.
>
> *sigh*. So does OS X Tiger. And (unsurprisingly), CentOS 4, which is
> what we use for all our Linux build servers.
>
> This is getting quite tiresome :-(

Much as I dislike it, we may need to revisit the idea about putting
the flex output files in CVS...


-- Magnus HaganderSelf: http://www.hagander.net/Work: http://www.redpill-linpro.com/

Magnus Hagander wrote:
> On Thu, Jul 23, 2009 at 10:52, Dave Page<dpage@pgadmin.org> wrote:
>   
>> On Thu, Jul 23, 2009 at 2:57 AM, Greg Smith<gsmith@gregsmith.com> wrote:
>>     
>>> I got bit by this tonight as part of testing a patch on CentOS 5, which like
>>> RHEL 5 still ships flex 2.5.4.  I just wrote a little guide on how to grab a
>>> source RPM from a Fedora version and install it to work around that problem:
>>> http://notemagnet.blogspot.com/2009/07/upgrading-flex-from-source-rpm-to.html
>>>
>>> Kind of annoying, but as special software you have to install on a server
>>> just to build something from CVS goes it's only a minor inconvenience.
>>>       
>> *sigh*. So does OS X Tiger. And (unsurprisingly), CentOS 4, which is
>> what we use for all our Linux build servers.
>>
>> This is getting quite tiresome :-(
>>     
>
> Much as I dislike it, we may need to revisit the idea about putting
> the flex output files in CVS...
>
>   

Why? This only affects developers building from a CVS pull. You don't 
need any flex at all to build from a tarball. If developers can't 
install flex on a *nix box they need to get out of the business.

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:
> Magnus Hagander wrote:
>> Much as I dislike it, we may need to revisit the idea about putting
>> the flex output files in CVS...

> Why? This only affects developers building from a CVS pull. You don't 
> need any flex at all to build from a tarball. If developers can't 
> install flex on a *nix box they need to get out of the business.

I wonder if it would be helpful to have a buildfarm option whereby
it would fetch the latest nightly-snapshot tarball and use that instead
of a CVS pull.  This would have the dual advantage of actually testing
builds from tarballs and requiring less stuff on the buildfarm machine.
It wouldn't be useful for more-than-once-a-day builds, but a lot of
the machines only build that often anyhow.
        regards, tom lane

Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>   
>> Magnus Hagander wrote:
>>     
>>> Much as I dislike it, we may need to revisit the idea about putting
>>> the flex output files in CVS...
>>>       
>
>   
>> Why? This only affects developers building from a CVS pull. You don't 
>> need any flex at all to build from a tarball. If developers can't 
>> install flex on a *nix box they need to get out of the business.
>>     
>
> I wonder if it would be helpful to have a buildfarm option whereby
> it would fetch the latest nightly-snapshot tarball and use that instead
> of a CVS pull.  This would have the dual advantage of actually testing
> builds from tarballs and requiring less stuff on the buildfarm machine.
> It wouldn't be useful for more-than-once-a-day builds, but a lot of
> the machines only build that often anyhow.
>
>             
>   

That is certainly doable. It would be in effect a forced run, because we 
would have no notion of what had changed. Presumably, this would only be 
for HEAD - we don't do daily snapshots of the back branches, do we?

cheers

andrew

Andrew Dunstan <andrew@dunslane.net> writes:
> Tom Lane wrote:
>> I wonder if it would be helpful to have a buildfarm option whereby
>> it would fetch the latest nightly-snapshot tarball and use that instead
>> of a CVS pull.

> That is certainly doable. It would be in effect a forced run, because we 
> would have no notion of what had changed. Presumably, this would only be 
> for HEAD - we don't do daily snapshots of the back branches, do we?

There was just recently some discussion of providing daily snapshots
for the most recent back branch or two.  I'm not sure if that's up
and running, but the consensus seemed to be that it was a good idea.
        regards, tom lane