Thread: Distribution making

Distribution making

From
Peter Eisentraut
Date:
I've been collecting a few ideas for integrating the distribution making
process into the build system. (I did take a look at the mk-release and
mk-snapshot scripts on hub.org as well.) I have a trial implementation
which works well, except that it doesn't build the documentation.

1. Instead of the release_prep script we have a new target `distprep'.
Those makefiles that want to build something for the distribution can
simply implement this target. This is nice because you have the distprep
and the correspondng distclean target in the same file. To make a
distribution-ready tree, you do

./configure
make distprep
make distclean

2. In order to get rid of the CVS directories (and anything else you don't
want to ship, like _deadcode) and get the directory name straight,
*without* clobbering your checked out tree, I create a directory
"postgresql-$(VERSION)" and copy the files I want to distribute in there.
The conceptual procedure is

# use your already configured tree
make distprep
mkdir postgresql-$(VERSION)
copy all files from . into postgresql-$(VERSION), expect those you don't want
make -C postgresql-$(VERSION) distclean

This is done by `make distdir'.

3. `make dist' depends on distdir, tars up the prepared tree, and leaves a
file postgresql-$(VERSION).tar.gz, which you can give to your friends.

4. If you got an extra half hour you can run `make distcheck', which:

* makes a distribution, using make dist
* unpacks the distribution
* runs configure
* runs make -q distprep, to check whether the files you just prepared for distribution are really still up to date
* builds and installs everything
* runs make uninstall and checks whether it really uninstalled everything
* makes another distribution from this test tree
* checks whether this distribution is sufficiently similar to the previous one (i.e., same files, same size)

This approach should guard against the common tarball making problems:
misnamed top-level directory, missing files, funky timestamps, etc.


If there's any interest in this I can commit it so you can take a look. It
doesn't affect anything else (including release_prep).

-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Re: Distribution making

From
Karel Zak
Date:
On Mon, 10 Jul 2000, Peter Eisentraut wrote:

> 2. In order to get rid of the CVS directories (and anything else you don't
> want to ship, like _deadcode) and get the directory name straight,
Need we still "_deadcode", if we have CVS? BTW. --- who compile backend
with "#define NOT_USED"?

> 3. `make dist' depends on distdir, tars up the prepared tree, and leaves a
> file postgresql-$(VERSION).tar.gz, which you can give to your friends.
And what ".orig.tar.gz"? We Debian's user love it :-) (and IMHO it is 
really not bad idea.)
                 Karel

PS. not flame please...



Re: Distribution making

From
Tom Lane
Date:
Sounds pretty good to me... but Marc's the guy who would normally be
using it, so it's his wishes you gotta cater to...
        regards, tom lane


Re: Distribution making

From
Peter Eisentraut
Date:
Karel Zak writes:

> > 3. `make dist' depends on distdir, tars up the prepared tree, and leaves a
> > file postgresql-$(VERSION).tar.gz, which you can give to your friends.
> 
>  And what ".orig.tar.gz"? We Debian's user love it :-) (and IMHO it is 
> really not bad idea.)

What's that?

-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Re: Distribution making

From
The Hermit Hacker
Date:
Sounds great to me ... what we have now is a major kludge ...

Basically, what I'd have to do for a release is CVSUP the latest code on
that branch, do a 'make dist' and that's it?  

Is this something that can be 'extended' to do the RPM also? :)

On Mon, 10 Jul 2000, Peter Eisentraut wrote:

> I've been collecting a few ideas for integrating the distribution making
> process into the build system. (I did take a look at the mk-release and
> mk-snapshot scripts on hub.org as well.) I have a trial implementation
> which works well, except that it doesn't build the documentation.
> 
> 1. Instead of the release_prep script we have a new target `distprep'.
> Those makefiles that want to build something for the distribution can
> simply implement this target. This is nice because you have the distprep
> and the correspondng distclean target in the same file. To make a
> distribution-ready tree, you do
> 
> ./configure
> make distprep
> make distclean
> 
> 2. In order to get rid of the CVS directories (and anything else you don't
> want to ship, like _deadcode) and get the directory name straight,
> *without* clobbering your checked out tree, I create a directory
> "postgresql-$(VERSION)" and copy the files I want to distribute in there.
> The conceptual procedure is
> 
> # use your already configured tree
> make distprep
> mkdir postgresql-$(VERSION)
> copy all files from . into postgresql-$(VERSION), expect those you don't want
> make -C postgresql-$(VERSION) distclean
> 
> This is done by `make distdir'.
> 
> 3. `make dist' depends on distdir, tars up the prepared tree, and leaves a
> file postgresql-$(VERSION).tar.gz, which you can give to your friends.
> 
> 4. If you got an extra half hour you can run `make distcheck', which:
> 
> * makes a distribution, using make dist
> * unpacks the distribution
> * runs configure
> * runs make -q distprep, to check whether the files you just prepared for
>   distribution are really still up to date
> * builds and installs everything
> * runs make uninstall and checks whether it really uninstalled everything
> * makes another distribution from this test tree
> * checks whether this distribution is sufficiently similar to the previous
>   one (i.e., same files, same size)
> 
> This approach should guard against the common tarball making problems:
> misnamed top-level directory, missing files, funky timestamps, etc.
> 
> 
> If there's any interest in this I can commit it so you can take a look. It
> doesn't affect anything else (including release_prep).
> 
> -- 
> Peter Eisentraut                  Sernanders v�g 10:115
> peter_e@gmx.net                   75262 Uppsala
> http://yi.org/peter-e/            Sweden
> 

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 



Re: Distribution making

From
Lamar Owen
Date:
The Hermit Hacker wrote:
> Is this something that can be 'extended' to do the RPM also? :)

Yes.  Do you want that to happen?  It won't be too hard -- the
requirements would be simply having an 'RPM' directory in the tarball
that would contain the spec file and the required patches and ancilliary
programs.  Then, you simply issue (on a RedHat system :-)) the
appropriate 'rpm -tb' (or 'rpm -ta') command. The binary RPM's will be
built to the usual place, and all is well with the world.

Although the spec file might need to be in the tarball's top-level
directory to work -- I'll have to investigate this.

So, the RPM-building would go like:
download postgresql-$version.tar.gz to %{_topdir}/SOURCES
(in %{_topdir}/SOURCES, execute 'rpm -tb' (or -ta).
Pick up your RPM's in %{_topdir}/RPMS/%{_arch}
Install them with rpm -Uvh.

No sweat.  See, the RPM building process is already highly automated --
once the patches are made, that is.

Although, I must admit -- it would be very nice to not have those
patches.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Re: Distribution making

From
The Hermit Hacker
Date:
On Mon, 10 Jul 2000, Lamar Owen wrote:

> The Hermit Hacker wrote:
> > Is this something that can be 'extended' to do the RPM also? :)
> 
> Yes.  Do you want that to happen?  It won't be too hard -- the
> requirements would be simply having an 'RPM' directory in the tarball
> that would contain the spec file and the required patches and ancilliary
> programs.  Then, you simply issue (on a RedHat system :-)) the
> appropriate 'rpm -tb' (or 'rpm -ta') command. The binary RPM's will be
> built to the usual place, and all is well with the world.
> 
> Although the spec file might need to be in the tarball's top-level
> directory to work -- I'll have to investigate this.
> 
> So, the RPM-building would go like:
> download postgresql-$version.tar.gz to %{_topdir}/SOURCES
> (in %{_topdir}/SOURCES, execute 'rpm -tb' (or -ta).
> Pick up your RPM's in %{_topdir}/RPMS/%{_arch}
> Install them with rpm -Uvh.
> 
> No sweat.  See, the RPM building process is already highly automated --
> once the patches are made, that is.
> 
> Although, I must admit -- it would be very nice to not have those
> patches.

If this can be worked in along side/on top of what Peter is doing, I think
that would be great ...




Re: Distribution making

From
Karel Zak
Date:
On Tue, 11 Jul 2000, Peter Eisentraut wrote:

> Karel Zak writes:
> 
> > > 3. `make dist' depends on distdir, tars up the prepared tree, and leaves a
> > > file postgresql-$(VERSION).tar.gz, which you can give to your friends.
> > 
> >  And what ".orig.tar.gz"? We Debian's user love it :-) (and IMHO it is 
> > really not bad idea.)
> 
> What's that?
This is the original source package. But 'tar.gz' is some common archive
file. See any debian's ftp....
                     Karel



Re: Distribution making

From
Karel Zak
Date:
On Mon, 10 Jul 2000, The Hermit Hacker wrote:

> Is this something that can be 'extended' to do the RPM also? :)
Into standard *source* tree? I was long time glad of PG's devevelopment,
that create dateless and OS independent source.
Well, we will have support for RPM, do you want support for Debian too?
If yes, do you accept /Debian directory in top source tree? 
Do you accept different directory philosophy for several dists.?

...etc.
IMHO to *source* tree belong to matter for binary making only. A
distribution must be out of source.
                                       Karel

PS. is it this equation right: "OS = Linux = RH"? I hope that not.



Re: Distribution making

From
"Oliver Elphick"
Date:
Peter Eisentraut wrote: >Karel Zak writes: > >> > 3. `make dist' depends on distdir, tars up the prepared tree, and
leaves    > a >> > file postgresql-$(VERSION).tar.gz, which you can give to your friends. >>  >>  And what
".orig.tar.gz"?We Debian's user love it :-) (and IMHO it is  >> really not bad idea.) > >What's that?
 

A Debian source package consists of the upstream source (named
package.orig.tar.gz) together with a diff file and a control file.  These
should be all that are needed to rebuild the binary package from the source.

Some upstream developers make it hard for us, though. :-(

-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
========================================   "I waited patiently for the LORD; and he inclined unto      me, and heard my
cry.He brought me up also out of an      horrible pit, out of the miry clay, and set my feet      upon a rock, and
establishedmy goings. And he hath      put a new song in my mouth, even praise unto our God.     Many shall see it, and
fear,and shall trust in the      LORD."                 Psalms 40:1-3 
 




Re: Distribution making

From
Lamar Owen
Date:
The Hermit Hacker wrote:
> On Mon, 10 Jul 2000, Lamar Owen wrote:
> > The Hermit Hacker wrote:
> > > Is this something that can be 'extended' to do the RPM also? :)

> > Yes.  Do you want that to happen?  It won't be too hard -- the

> If this can be worked in along side/on top of what Peter is doing, I think
> that would be great ...

Well, it can be done.  However, the RPM' have historically had a tighter
cycle than the official release -- We have 7.0.2-1 and -2 RPM's (soon to
be up to -6 or -7) a problems are found in the packaging. Of course, as
I try to synchronize things, maybe that cycle time can go away. :-)

Now, I know that, in the past, feature patches were released on
postgresql.org (as well as bugfix patches) that were inbetween releases
-- that could be done for the RPM stuff easily enough.  But, I'd want
some comments from RPM distribution users before going that route.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Re: Distribution making

From
Lamar Owen
Date:
Karel Zak wrote:
> On Mon, 10 Jul 2000, The Hermit Hacker wrote:
> > Is this something that can be 'extended' to do the RPM also? :)
>  Into standard *source* tree? I was long time glad of PG's devevelopment,
> that create dateless and OS independent source.
>  Well, we will have support for RPM, do you want support for Debian too?
> If yes, do you accept /Debian directory in top source tree?
> Do you accept different directory philosophy for several dists.?

I don't mind one bit to have multiple distribution support in the
tarball.  
> ...etc.
>  IMHO to *source* tree belong to matter for binary making only. A
> distribution must be out of source.

There are more files in the RPM set, for instance, than just the
binaries -- there is the spec file, which controls the building of the
RPM's; there is a patch file (to patch around some madness in the source
that breaks the RPM package in some way); there are a couple of scripts
(startup and upgrade); there is a man page for  the upgrade script; as
well as other things.  Now, getting the *source* of the RPM distribution
packaging into the tarball might be OK -- I'm certainly not advocating
packaging binaries in the tarball!

RPM is used on many more systems than just RedHat Linux -- and I am
actively trying to get the *source* RPM to be buildable on as many
RPM-based systems as possible -- including being buildable on a Solaris
system that might have RPM installed, or a Tru64 system with RPM
installed (they DO exist).  Unfortunately there are a number of
RedHat-isms in the current RPMset that are having to be worked around --
although, this situation promises to improve in the near future, if I
have anything to do with it.

Therefore, a separate source RPM would not need to be distributed -- a
person can just download the tarball, execute a single command, and have
properly built RPM's for their system ready to install.  Can't get much
easier than that!

> PS. is it this equation right: "OS = Linux = RH"? I hope that not.

No, and neither is RPM == "RedHat Linux Only."

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Re: Distribution making

From
Karel Zak
Date:
On Tue, 11 Jul 2000, Lamar Owen wrote:

> >  IMHO to *source* tree belong to matter for binary making only. A
> > distribution must be out of source.
> 
> There are more files in the RPM set, for instance, than just the
> binaries -- there is the spec file, which controls the building of the
> RPM's; there is a patch file (to patch around some madness in the source
> that breaks the RPM package in some way); there are a couple of scripts
> (startup and upgrade); there is a man page for  the upgrade script; as
> well as other things.  Now, getting the *source* of the RPM distribution
> packaging into the tarball might be OK -- I'm certainly not advocating
> packaging binaries in the tarball!
I good known how act RH packaging. And I probably understant you.

> Therefore, a separate source RPM would not need to be distributed -- a
> person can just download the tarball, execute a single command, and have
> properly built RPM's for their system ready to install.  Can't get much
> easier than that!

Yes, I know. But it expect that in the *common-original-source* must be
.spec file. Or not? 
I'm not enemy of RH, I only not sure if is good "foul" original source.
                Karel 




Re: Distribution making

From
Peter Eisentraut
Date:
The Hermit Hacker writes:

> Basically, what I'd have to do for a release is CVSUP the latest code on
> that branch, do a 'make dist' and that's it?  

Yes. You'll just have to copy the documentation files in there first. Of
course eventually that should work better as well.


> Is this something that can be 'extended' to do the RPM also? :)

I'd have to look at how the RPMs are made, but in theory why not.


-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Re: Distribution making

From
Pavel.Janik@linux.cz (Pavel Janík ml.)
Date:
From: Karel Zak <zakkr@zf.jcu.cz>  Date: Tue, 11 Jul 2000 18:22:30 +0200 (CEST)

Hi Karel,
  > Yes, I know. But it expect that in the *common-original-source* must  > be .spec file. Or not?

not, of course. But if it is there, you can easily built RPM via rpm -ta
source*tar.gz. Only Source RPMS (aka SRPM) must have .spec file built in.
  >  I'm not enemy of RH, I only not sure if is good "foul" original  >  source.

The same for me. I do hate packagers who just rename my package.tar.gz to
package.orig.tar.gz and built tens of patches into one big .diff.gz which
is crazy to maintain :-)) No flamewars please.

BTW - Searching of mailing-lists is not working on postgresql.org. Try this
very long link :-))


http://www.postgresql.org/mhonarc/pgsql-hackers/search.cgi?ps=10&ul=http%3A%2F%2Fwww.postgresql.org%2Fmhonarc%2Fpgsql-hackers%2F%25&q=CREATE+SNAPSHOT&ps=10&o=0&m=all
-- 
Pavel Janík ml.
Pavel.Janik@linux.cz