Re: Packaging automation, testing and work-reduction - Mailing list pgsql-pkg-yum

On 08/20/2014 02:01 AM, Jeff Frost wrote:
> This all sounds exciting, but since Devrim is out, let's table the discussion till he gets back (or at least gets
neara keyboard). 

Devrim, any thoughts re the following?

I'm particularly interested in adopting a unified specfile for the
postgresql package (for starters), then adopting mock for builds. With
Koji and packages in git to follow.

> On Aug 19, 2014, at 1:45 AM, Craig Ringer <craig@2ndQuadrant.com> wrote:
>
>> Hi all
>>
>> I'm interested in getting more involved in the RPM packaging process for
>> PostgreSQL.  I know I'm new to the scene, so the following might be
>> jumping the gun a bit, but I'm also aware that it's hard to find time
>> for PGDG RPM work and more contribution might be valued.
>>
>> I've produced new spec files for PostgreSQL 9.4 that unify all of
>> EL-5/6/7 and Fedora 19/20/21 into a single spec that can be rebuilt for
>> each platform, and I think that's ready for wider testing now. Builds
>> for all platforms work correctly using 'mock' on a Fedora 20 build-host,
>> and they test-install into the target platform correctly, and
>> interoperate with existing PGDG RPMs.
>>
>> I've fixed a number of packaging bugs in the process. I'll attach the
>> spec and the rest of the RPM sources in a followup mail to the "unifying
>> the spec files" thread.
>>
>> I'm looking at future steps now, and I'd like to outline some ideas that
>> might make PGDG RPMs easier to maintain, more friendly toward
>> contribution from others, more reliable, easier to test, and more
>> inter-operable with other repositories.
>>
>> My personal motivation - just so I'm upfront about this - is that what
>> I'm talking about would make it a *lot* easier for me to produce
>> "hotfix" RPMs for 2ndQuadrant customers who want fixes backported before
>> a new minor release, produce backport RPMs where they want something
>> backported into 9.2 or whatever, produce RPMs for customers with
>> perf/bug fixes that haven't been accepted by upstream yet, and make
>> packaging BDR a lot easier too.
>>
>> From my understanding of how things currently work I think it'd also
>> make it easier to maintain PGDG, but of course I don't have much history
>> here or the experience with the current process to really back that.
>>
>>
>> I'd like to (short version):
>>
>> * Depend on EPEL
>>  (some packages already do, it's just not declared)
>> * Drop packages from PGDG that're already in EPEL
>> * Unify spec files for different dists
>> * Build with Mock
>> * Use Koji
>> * Test with Mock
>> * Let Koji handle createrepo etc
>> * Move package management to git
>>
>>
>> Details of what I'm suggesting:
>>
>> - Depend on EPEL for RHEL/CentOS/SciLinux and remove all packages
>>  provided by EPEL from PGDG to reduce duplication and potential for
>>  conflicts. (This might involve applying for EPEL dev status
>>  eventually, but shouldn't to start with).
>>
>>  As many packages would work w/o EPEL I doubt it'd have to be a hard
>>  dependency. Just say that EPEL is recommended and will be required
>>  for some packages.
>>
>>
>> - Progressively unify spec files for other packages to remove the need
>>  for individual specs for each target OS, like I've already done
>>  for postgresql94. (Done for Pg 9.4, not prior, but see next item).
>>
>>
>> - For PostgreSQL packages also make the major versions supported in the
>>  same define, just by changing the macros. So packaging bug fixes are
>>  easier. (I haven't done this, but it doesn't look too hard).
>>
>>
>> - Always build packages using mock, so issues with build-depends and
>>  undeclared runtime dependencies like those I've recently reported are
>>  identified at build time. (This works with the packages I've
>>  rewritten).
>>
>>  Use mock target definition files that include EPEL repositories for
>>  EL targets.
>>
>>
>> - Test-install packages in another mock chroot after building, and
>>  run some basic tests - initdb, etc, at least. Trivial to automate,
>>  just run a script within the mock chroot. (I've done part of this
>>  and it's not hard to finish off).
>>
>>
>> - Automate package building with mock via Koji by setting up a Koji
>>  server for PGDG. This is fairly well documented, and really just
>>  needs a host with a pile of disk space, preferably running a new
>>  Fedora release. Builds can then be submitted to Koji, either as SRPMs
>>  or via pushes to git. The latter involves tarballs in git, but
>>  that's what Fedora does.
>>
>>  Koji can pull the packages from Fedora and EPEL to use as a base
>>  for builds, but (using Mock) will only install and enable those
>>  declared for dependencies or build-depends in any individual build.
>>
>>  I haven't set a Koji up, but I'd very much like to. If PGDG
>>  wants to go this way I'll be happy to do it for PGDG, rather than
>>  creating a private Koji.
>>
>>  Check out:
>>
>>  http://koji.fedoraproject.org/koji/
>>  https://fedorahosted.org/koji/
>>
>>  'mash' is used to manage koji -> yum repo work.
>>
>>
>> - Let Koji take care of running createrepo, repoview, etc as updates
>>  are pushed, rather than running the manually;
>>
>>
>> - Test repositories with repoclosure and complain if we find
>>  dependencies that can't be satisfied, e.g.
>>
>>    repoclosure -l fedora -l pgdg94
>>
>>  or
>>
>>    repoclosure -l el5 -l epel5 -l pgdg91
>>
>>  (with a corresponding yum configuration).
>>
>>  This can be done within 'mock', or with individual yum config files
>>  rather than using the system ones.
>>
>>
>>
>> Thoughts?
>>
>> It probably makes sense to fire up a Koji instance first. Then start
>> progressively getting packages building with corrected build-depends and
>> depends, using mock, and where possible unified to build the same spec
>> on all dists, moving them into Koji-friendly git management as that happens.
>>
>> As packages are updated, the copy in svn would want to be updated to
>> match so there's no drift, marking the spec in git as the authoritative one.
>>
>> Once everything's building happily in Koji its repos could become the
>> authorative ones.
>>
>> Crazy talk?
>> --
>> Craig Ringer                   http://www.2ndQuadrant.com/
>> PostgreSQL Development, 24x7 Support, Training & Services
>>
>>
>> --
>> Sent via pgsql-pkg-yum mailing list (pgsql-pkg-yum@postgresql.org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-pkg-yum
>
> ---
> Jeff Frost <jeff@pgexperts.com>
> CTO, PostgreSQL Experts, Inc.
> Phone: 1-888-PG-EXPRT x506
> FAX: 415-762-5122
> http://www.pgexperts.com/
>
>
>
>
>


--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services