Thread: New and unified 9.5 spec file is in git
Hi, After various attempts to make RHEL 5 happy, I pushed unified PostgreSQL 9.5 spec file to git, which is supposed to work for all distros. The spec file also adds plpython3 subpackage (on supported distros), including patches from Fedora for Python 3.5 tests (on supported distros), adds dtrace support (on supported distros), and some cosmetic cleanup. I'm about to push packages to repo. I tested them, hopefully I did not miss anything :-) If anyone could take a look at it, it would be awesome. Regards, -- Devrim GÜNDÜZ Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer Twitter: @DevrimGunduz , @DevrimGunduzTR
Attachment
On 20 January 2016 at 05:44, Devrim GÜNDÜZ <devrim@gunduz.org> wrote:
Hi,
After various attempts to make RHEL 5 happy, I pushed unified
PostgreSQL 9.5 spec file to git, which is supposed to work for all
distros.
Great! That's wonderful news.
I hope you got some use out of the work I sent you a while ago on that. Apologies that I didn't find the time to follow up on it.
If anyone could take a look at it, it would be awesome.
I'll look right now.
On 20 January 2016 at 10:15, Craig Ringer <craig@2ndquadrant.com> wrote:
If anyone could take a look at it, it would be awesome.I'll look right now.
To make it clear that they're the same is it worth having a common/ directory at the same level as EL-5, F-23, etc and symlinking the specfile to the various subdirs from there, along with any identical patches etc?
Here's the full build process I used:
sudo yum -y install rpmdevtools mock
git clone git://git.postgresql.org/git/pgrpms.git
cd pgrpms/rpm/redhat/9.5/postgresql/F-23
spectool --get-files postgresql-9.5.spec
mock -r fedora-23-x86_64 --buildsrpm --sources . --spec postgresql-9.5.spec
cp /var/lib/mock/fedora-23-x86_64/result/postgresql95-9.5.0-2PGDG.fc23.src.rpm .
mock -r fedora-23-x86_64 --rebuild postgresql95-9.5.0-2PGDG.fc23.src.rpm
I'm really happy about this, since it means the build depends are now correct and it can be built without needing a dedicated build machine using nothing but mock. No need to manually set up dependencies anymore, set up a build box, etc. No need to juggle x86_64 and i386 either, you can run builds for *everything* from a single F23 box or VM or docker or whatever.
If the same process is followed for other packages then it'll become possible to build the whole PGDG system with mockchain with no custom build machines, building anywhere that mock is available. No accidental undeclared build or runtime dependencies. Since you seem to be using Docker to prep build envs now it's less of an issue than when the build boxes are hand-maintained, but the great thing about mock is that it gets all the build depends info straight from the spec file.
I also rebuilt the F23 package for F21 just by changing the -r argument to mock. Same for building it for i386 on an x86_64 host.
The EL-5 package built happily under mock for -r epel-5-i386 too. I didn't test install it.
Thanks so much. This is great and I really hope you got some use out of my prior work on it.
I've uploaded a branch with a couple of small improvements. See https://github.com/2ndQuadrant/pgdg-pgrpms .
git remote add 2ndq https://github.com/2ndQuadrant/pgdg-pgrpms.git
git fetch 2ndq
# view it
git log -p remotes/2ndq/add-builddepends
# merge it
git merge remotes/2ndq/add-builddepends
This is the main patch of interest:
A further suggested improvement, which I haven't put in that tree but wanted to flag as an idea - detect the target based on the directory name so there's no need to have a different Makefile for each target dir and they can all just be symlinks.
For each dist's Makefile:
-DIST=.rhel5
SPECFILE="postgresql-9.5.spec"
+# This gibberish gets the immediate parent directory's name, prepends RPMDIST_
+# and looks it up as a Make variable lazily on first evaluation. That way we
+# don't require a new Makefile for each build directory.
+#
+# The RPMDIST_ vars are defined in Makefile.global
+#
+DIST=$(RPMDIST_$(shell basename $(dir $(abspath $(lastword $(MAKEFILE_LIST))))))
then in Makefile.global:
+# These mappings could be automated with $(foreach ...) and $(eval ...)
+# but there hardly seems any point.
+RPMDIST_EL-5=.rhel5
+RPMDIST_EL-6=.rhel6
+RPMDIST_EL-7=.rhel7
+RPMDIST_F-21=.fc21
+RPMDIST_F-22=.fc22
+RPMDIST_F-23=.fc23
Then EL-5, EL-6, EL-7, F-22 and F-23 can be deleted and replaced with common/ which is symlinks to each of them. That shouldn't upset the build system; otherwise I'd suggest switching moving all files up a level, removing the dist subtrees, and using "make DIST=EL-5" or whatever to control the target.
Less duplication in the source tree, no need to remember to copy files around, easier to track history.
On 20 January 2016 at 10:15, Craig Ringer <craig@2ndquadrant.com> wrote:If anyone could take a look at it, it would be awesome.I'll look right now.It built perfectly with mock on Fedora 21 for a F23 target. Brilliant!To make it clear that they're the same is it worth having a common/ directory at the same level as EL-5, F-23, etc and symlinking the specfile to the various subdirs from there, along with any identical patches etc?Here's the full build process I used:sudo yum -y install rpmdevtools mockgit clone git://git.postgresql.org/git/pgrpms.gitcd pgrpms/rpm/redhat/9.5/postgresql/F-23spectool --get-files postgresql-9.5.specmock -r fedora-23-x86_64 --buildsrpm --sources . --spec postgresql-9.5.speccp /var/lib/mock/fedora-23-x86_64/result/postgresql95-9.5.0-2PGDG.fc23.src.rpm .mock -r fedora-23-x86_64 --rebuild postgresql95-9.5.0-2PGDG.fc23.src.rpm
I'm really happy about this, since it means the build depends are now correct and it can be built without needing a dedicated build machine using nothing but mock. No need to manually set up dependencies anymore, set up a build box, etc. No need to juggle x86_64 and i386 either, you can run builds for *everything* from a single F23 box or VM or docker or whatever.If the same process is followed for other packages then it'll become possible to build the whole PGDG system with mockchain with no custom build machines, building anywhere that mock is available. No accidental undeclared build or runtime dependencies. Since you seem to be using Docker to prep build envs now it's less of an issue than when the build boxes are hand-maintained, but the great thing about mock is that it gets all the build depends info straight from the spec file.I also rebuilt the F23 package for F21 just by changing the -r argument to mock. Same for building it for i386 on an x86_64 host.The EL-5 package built happily under mock for -r epel-5-i386 too. I didn't test install it.Thanks so much. This is great and I really hope you got some use out of my prior work on it.
This makes me so excited! I can't wait to try it and work on getting the other packages set up this way as well!
Great job you guys!
--
Hi Craig, On Wed, 2016-01-20 at 10:15 +0800, Craig Ringer wrote: > I hope you got some use out of the work I sent you a while ago on > that. Apologies that I didn't find the time to follow up on it. Even though I did not use your actual work, I must admit that I learned *a lot* from your spec. Apologies for not crediting you over there. Thank you very much! Regards, -- Devrim GÜNDÜZ Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer Twitter: @DevrimGunduz , @DevrimGunduzTR
Attachment
Hi, On Wed, 2016-01-20 at 11:52 +0800, Craig Ringer wrote: > It built perfectly with mock on Fedora 21 for a F23 target. > Brilliant! Thanks for checking! > To make it clear that they're the same is it worth having a common/ > directory at the same level as EL-5, F-23, etc and symlinking the > specfile to the various subdirs from there, along with any identical > patches etc? That would be the next step. Let's finish the actual work first. > I'm really happy about this, since it means the build depends are now > correct and it can be built without needing a dedicated build machine > using nothing but mock. No need to manually set up dependencies > anymore, set up a build box, etc. No need to juggle x86_64 and i386 > either, you can run builds for *everything* from a single F23 box or > VM or docker or whatever. Well, we are building 100+ packages in 9.5 set. Setting up a new mock environment everytime would slow us down, IMHO. What do you think? > If the same process is followed for other packages *All* of the *recently added* spec files are unified nowadays :) This also applies to some specs that I found time to rework on (like postgis, repmgr, etc.). The repo is not fully unified yet, but we are on it. > Since you seem to be using Docker to prep build envs now Err, no. We are using actual VMs (powered by libvirtd+qemu-kvm) > it's less of an issue than when the build boxes are hand-maintained, > but the great thing about mock is that it gets all the build depends > info straight from the spec file. All deps are already installed on our build servers. If not, we install them before building a new package. > The EL-5 package built happily under mock for -r epel-5-i386 too. I > didn't > test install it. This is great testing. Appreciated, thanks! > Thanks so much. This is great and I really hope you got some use out > of my prior work on it. You are welcome. As I wrote a few mins ago, I really was inspired from your work. Regards, -- Devrim GÜNDÜZ Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer Twitter: @DevrimGunduz , @DevrimGunduzTR
Attachment
Hi, On Wed, 2016-01-20 at 12:22 +0800, Craig Ringer wrote: > This is the main patch of interest: > > https://github.com/2ndQuadrant/pgdg-pgrpms/commit/60524156129b27b9cf3 > 836621a0d05afb658d60e Oh, why do we need -devel packages for postgresql95-devel package? Regards, -- Devrim GÜNDÜZ Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer Twitter: @DevrimGunduz , @DevrimGunduzTR
Attachment
Hi Jeff, On Tue, 2016-01-19 at 20:33 -0800, Jeff Frost wrote: > This makes me so excited! I can't wait to try it and work on getting > the other packages set up this way as well! > > Great job you guys! Thanks! Regards, -- Devrim GÜNDÜZ Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer Twitter: @DevrimGunduz , @DevrimGunduzTR
Attachment
On 21 January 2016 at 04:18, Devrim GÜNDÜZ <devrim@gunduz.org> wrote:
> This is the main patch of interest:
>
> https://github.com/2ndQuadrant/pgdg-pgrpms/commit/60524156129b27b9cf3
> 836621a0d05afb658d60e
Oh, why do we need -devel packages for postgresql95-devel package?
So that when you install the postgresql95-devel package it's reliably usable.
If we don't depend on gettext-devel etc, then where PostgreSQL uses those headers in its own headers you'll get errors at compile time unless they happen to be installed because of the requirements of other unrelated packages.
On 21 January 2016 at 04:17, Devrim GÜNDÜZ <devrim@gunduz.org> wrote:
> I'm really happy about this, since it means the build depends are now
> correct and it can be built without needing a dedicated build machine
> using nothing but mock. No need to manually set up dependencies
> anymore, set up a build box, etc. No need to juggle x86_64 and i386
> either, you can run builds for *everything* from a single F23 box or
> VM or docker or whatever.
Well, we are building 100+ packages in 9.5 set. Setting up a new mock
environment everytime would slow us down, IMHO. What do you think?
mock caches the build env so in practice it's very fast. mockchain helps even more.
mock is what Koji uses and what's used to build all of Fedora and EPEL, so it gets a fair bit of performance attention. There _is_ a performance cost vs building bare on a preconfigured build box but it's (IMO) greatly offset by the increased reliability and consistency of package builds. You don't risk having undeclared dependencies, building against a version other than the one declared in the spec file, etc.
> If the same process is followed for other packages
*All* of the *recently added* spec files are unified nowadays :)
Great!
> it's less of an issue than when the build boxes are hand-maintained,
> but the great thing about mock is that it gets all the build depends
> info straight from the spec file.
All deps are already installed on our build servers. If not, we install
them before building a new package.
That can cause issues though. For example I think that's how the Java 8 problems a while ago happened. The spec file declared a dependency on openjdk 7 but the rpms were built against what was locally installed, which is openjdk 8.
The nice thing with mock is that the buildroot can contain only exactly the required packages for a build. No extras. It provides a lot more confidence that the spec matches the actual build and that anyone can reproduce the package build consistently without a special hand-created build env.
I've been using it for the BDR and pglogical builds and it does a pretty good job. It's helped me find a number of bugs and oversights too.