Thread: De-duplicating the tree
Hi all
--
Now that more and more packages share spec files and often more across the different OS versions, if not (yet) across Pg versions, I think it's worth looking at how to reduce the manual copy&paste involved in doing simple updates. It's error prone and tedious, and it shouldn't really be necessary.
Since I presume the build scripts etc are pretty heavily reliant on the current tree structure and there's a lot of legacy that might still need rebuilding one day, I'd like to symlink everything shared so that the build system doesn't get upset. I've checked that mock won't be upset, you just pass --symlink-dereference when building the srpm. If symlinks upset whatever the build scripts are used for the main repo tree it should be pretty trivial to tweak them to make them follow symlinks.
I'm thinking of starting with moving specs etc into per-version common/ dirs, like
rpm/redhat/9.5/repmgr/EL-6/repmgr.spec
=> rpm/redhat/9.5/repmgr/common/repmgr.spec
That'll bring the number of changes needed for many packages from 20+ down to 3-5 for a common update.
I'll prep a branch for, say, repmgr showing how it'd look so you can test your build scripts.
On 3 March 2016 at 21:45, Craig Ringer <craig@2ndquadrant.com> wrote:
That'll bring the number of changes needed for many packages from 20+ down to 3-5 for a common update.
BTW, if that works out I think the next step would be to template the specfiles, substituting the pg version into them. So we could then have a *single* spec file for all supported versions, at the cost of having to do some simple template substitutions on specs. This is what I've done very successfully in 2ndQuadrant's repositories and build scripts, so I can build things like pglogical for all supported Pg versions on all supported arches and hosts with a *single* command, from a *single* spec file, complete with rpm signing etc.
RPM macros aren't really suitable for this because rpm offers no way to embed macros in a .srpm, so you need to know the extra macros to --define at rpmbuild --rebuild time to get the expected result. Not desirable, a srpm should be self-contained. Hence templating the spec files.
For files that're reused across all supported versions as well, we could symlink like:
rpm/redhat/9.5/repmgr/EL-6/repmgr.spec.in
=> rpm/redhat/common/repmgr/repmgr.spec.in
where repmgr.spec.in is the existing specfile that simply has:
%global pgmajorversion __PG_MAJOR_SHORT__
%global pgpackageversion __PG_MAJOR__
There's already a Makefile, so adding rules for templated files is trivial and can be done via Makefile.global to make it a one-liner like
PG_VERSION_SUBST := repmgr.in repmgr.conf.sample.patch.in
The rules we'd generate would be like:
repmgr.spec: repmgr.spec.in
sed -e 's/__PG_MAJOR_SHORT__/$(pg_major_short)/' \
-e 's/__PG_MAJOR__/$(pg_major)/' \
$< > $@
the makevars pg_major_short and pg_major can be determined automatically from the makefile's path so there's no need to put the version in the Makefile.
(BTW, the DIST that's currently set manually in each Makefile can also be determined from the makefile path automatically)
Then I can probably patch Makefile.global to allow use of mock/mockchain just by passing USE_MOCK=1 to make, referencing a localrepo for build dependencies, so it becomes possible for anyone to rebuild anything in the source tree and expect it to work. That in turn leads to easy automated testing...
Sound like a sane direction? I don't want to tackle all that at once - for one thing, time is a concern. But starting with symlinking into common dirs *within* a pg major version and maybe autodetecting DIST so the Makefile can be shared would be a beginning.
Hi, Sure, please go ahead. While you are working on it, can you please also implement the feature we discussed offlist, which eliminates the pgmajorversion variables to be removed from the top of all spec files? I'm happy to change build scripts to cope with this. Regards, Devrim On Thu, 2016-03-03 at 21:45 +0800, Craig Ringer wrote: > Hi all > > Now that more and more packages share spec files and often more across the > different OS versions, if not (yet) across Pg versions, I think it's worth > looking at how to reduce the manual copy&paste involved in doing simple > updates. It's error prone and tedious, and it shouldn't really be necessary. > > Since I presume the build scripts etc are pretty heavily reliant on the > current tree structure and there's a lot of legacy that might still need > rebuilding one day, I'd like to symlink everything shared so that the build > system doesn't get upset. I've checked that mock won't be upset, you just > pass --symlink-dereference when building the srpm. If symlinks upset > whatever the build scripts are used for the main repo tree it should be > pretty trivial to tweak them to make them follow symlinks. > > I'm thinking of starting with moving specs etc into per-version common/ > dirs, like > > rpm/redhat/9.5/repmgr/EL-6/repmgr.spec > => rpm/redhat/9.5/repmgr/common/repmgr.spec > > That'll bring the number of changes needed for many packages from 20+ down > to 3-5 for a common update. > > I'll prep a branch for, say, repmgr showing how it'd look so you can test > your build scripts. > -- Devrim GÜNDÜZ Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer Twitter: @DevrimGunduz , @DevrimGunduzTR
Attachment
Hi, On Thu, 2016-03-03 at 21:58 +0800, Craig Ringer wrote: > > (BTW, the DIST that's currently set manually in each Makefile can also be > determined from the makefile path automatically) Quick note: The dir names are: EL-6, EL-7, for example -- but the dist variables are rhel6 and rhel7, respectively. We should keep the latter, for sure. I'm open for changing the dir names, or invent a way to translate EL-6 to rhel6. Regards, -- Devrim GÜNDÜZ Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer Twitter: @DevrimGunduz , @DevrimGunduzTR