Thread: get rid of distprep?

I'm thinking about whether we should get rid of the distprep target, the 
step in the preparation of the official source tarball that creates a 
bunch of prebuilt files using bison, flex, perl, etc. for inclusion in 
the tarball.  I think this concept is no longer fitting for contemporary 
software distribution.

There is a lot of interest these days in making the artifacts of 
software distribution traceable, for security and legal reasons.  You 
can trace the code from an author into Git, from Git into a tarball, 
somewhat from a tarball into a binary package (for example using 
reproduceable builds), from a binary package onto a user's system. 
Having some mystery prebuilt files in the middle there does not feel 
right.  Packaging guidelines nowadays tend to disfavor such practices 
and either suggest, recommend, or require removing and rebuilding such 
files.  This whole thing was fairly cavalier when we shipped gram.c, 
scan.c, and one or two other files, but now the number of prebuilt files 
is more than 100, not including the documentation, so this is a bit more 
serious.

Practically, who even uses source tarballs these days?  They are a 
vehicle for packagers, but packagers are not really helped by adding a 
bunch of prebuilt files.  I think this practice started before there 
even were things like rpm.  Nowadays, most people who want to work with 
the source should and probably do use git, so making the difference 
between a git checkout and a source tarball smaller would probably be 
good.  And it would also make the actual tarball smaller.

The practical costs of this are also not negligible.  Because of the 
particular way configure handles bison and flex, it happens a bunch of 
times on new and test systems that the build proceeds and then tells you 
you should have installed bison 5 minutes ago.   Also, extensions cannot 
rely on bison, flex, or perl being available, except it often works so 
it's not dealt with correctly.

Who benefits from these prebuilt files?  I doubt anyone actually has 
problems obtaining useful installations of bison, flex, or perl.  There 
is the documentation build, but that also seems pretty robust nowadays 
and in any case you don't need to build the documentation to get a 
useful installation.  We could make some adjustments so that not 
building the documentation is more accessible.  The only users of this 
would appear to be those not using git and not using any packaging. 
That number is surely not zero, but it's probably very small and doesn't 
seem worth catering to specifically.

Thoughts?

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> I'm thinking about whether we should get rid of the distprep target, ...

> Who benefits from these prebuilt files?  I doubt anyone actually has 
> problems obtaining useful installations of bison, flex, or perl.

I'm sure it was a bigger issue twenty years ago, but yeah, nowadays
our minimum requirements for those tools are so ancient that everybody
who cares to build from source should have usable versions available.

I think the weak spot in your argument, though, is the documentation.
There is basically nothing that is standardized or reproducible in
that toolchain, as every platform names and subdivides the relevant
packages differently, if they exist at all.  I was reminded of that
just recently when I updated my main workstation to RHEL8, and had to
jump through a lot of hoops to get everything installed that's needed
to build the docs (and I still lack the tools for some of the weirder
products such as epub).  I'd be willing to say "you must have bison,
flex, and perl to build" --- and maybe we could even avoid having a
long discussion about what "perl" means in this context --- but I
fear the doc tools situation would be a mess.

> The only users of this 
> would appear to be those not using git and not using any packaging. 

No, there's the packagers themselves who would be bearing the brunt of
rediscovering how to build the docs on their platforms.  And if the
argument is that there's a benefit to them of making the build more
reproducible, I'm not sure I buy it, because of (1) timestamps in the
output files and (2) docbook's willingness to try to download missing
bits off the net.  (2) is a huge and not very obvious hazard to
reproducibility.

But maybe you ought to be surveying -packagers about the question
instead of theorizing here.  Would *they* see this as a net benefit?

One other point to consider is that distprep or no distprep, I'd be
quite sad if the distclean target went away.  That's extremely useful
in normal development workflows to tear down everything that depends
on configure output, without giving up some of the more expensive
build products such as gram.c and preproc.c.

            regards, tom lane