Hackathon news - Mailing list pgsql-pkg-debian

Hi,

before too much time has passed, let's get activity back to this
mailing list.

Dimitri, Magnus and I had some semi-secret meeting at my house, and
worked out the blueprint for getting things flying. I'll just post our
notes and my blog post here as well so people have a chance to catch
up and don't have to dig into various places.

https://github.com/dimitri/apt.postgresql.org/blob/master/hackaton-20120821.md

-----------

# apt.postgresql.org hackathon

The hackathon happened in Credativ place and Christoph Berg and included
Magnus Hagander and Dimitri Fontaine. It took place 2012-08-21 and 22.

That document is both a report of what we managed to do in the hackathon,
including some decisions we took, and a TODO list for later.

## Hackathon goal

We want to build all PostgreSQL stable releases for all supported debian and
ubuntu distributions for i386 and amd64. We already have a Jenkins setup
allowing us to manage a build queue, thanks to Christoph.

## Building pl/sh for squeeze

We need to:

 1. get our own postgresql-common package with the right supported-versions script
 2. change the cowbuilder image source list to inclure our packages
 3. build plsh in the build environment
 4. get plsh packages for squeeze for all those PG versions
 5. publish them in a debian repository in localhost

### postgresql-common

Editing supported-versions is enough, `postgresql-server-dev-all`
dependencies are taken from that at build time.

### apt preferences

We want that the `~pgdg` packages take priority over the debian provided
packages for the same versions, so we need to do some APT pinning magic here
to make that happen, and include that setup in the repository and in the
installation instructions.

Default policy from the repository is a backports like policy.

We also provide a package that install a
`/etc/apt/preferences.d/apt.postgresql.org` file with the right pinning for
a full pgdg experience:

    Package: *
    Pin: origin apt.postgresql.org
    Pin-Priority: 600

### libpq

We ship latest stable PostgreSQL version of libpq in all stable
distributions (that's libpq from 9.1 for squeeze and wheezy), and we ship
the latest-including-beta in sid (libpq from 9.2 nowadays).

We need to build extensions with that libpq version too, of course.

### distributions and release policy

We publish maintained PostgreSQL versions to current debian and ubuntu
releases. We also publish old PostgreSQL versions to current and old debian
and ubuntu versions.

Distributions examples with wheezy:

 - wheezy-pgdg               (only PG supported versions, 8.3 -> 9.1)
 - wheezy-pgdg-testing       (including 9.2)
 - wheezy-pgdg-deprecated    (including 8.1)

Same thing with lenny, squeeze, jessy(?), sid, and ubuntu releases lucid,
precise and some others.

# Problems to solve later

## Automatic build

Several sources are available for the packaging. We can either build from
the current `apt-get source` packaging found in sid or from the source code
repository on `alioth` (svn and git are in use over there).

One idea is to build *testing* packages from their source code repositories
and *production* package directely from the `sid` distribution. That means
we're doing both Quality Assurance and Backports, but that might be a little
too much for the first version of this build system.

## Build Images Optimisation

It should be possible to provide for build images (pbuilder, cowbuilder)
including PostgreSQL build dependancies and our own packages pre-installed.
That means we need to rebuild those image at each PostgreSQL Minor release.

We might need a Jenkins job to do that.

## apt.postgresql.org website

We need some templates and build the website with them, or just use django
here. The main installation instruction should go to the main PostgreSQL
website. We agree that Magnus will do the django parts.

## Where to get the sources from

The current Jenkins install is fetching package sources from `bzr`, `git`
and `svn` repositories, we could do it the *backports* way and fetch debian
sid sources instead with `apt-get source`.

## pg_buildext

`debian/pgversions` should include a `*` facility so that we can build
extensions against new PostgreSQL releases without any editing. Worst case
it doesn't work and we report a bug.

## publishing policy

Generally we want to only include packages that come from Debian unstable,
usually in the version from there. Exceptions should be rare so we don't run
out of sync and lose track.

We need to publish a policy about Software Licences and content (postgresql,
extensions, some special cases for clients, but not any client software). We
only publish backend software with an OSI approved licence.

# Build Environment

## Sources

- PostgreSQL common packaging is on bzr
  http://anonscm.debian.org/bzr/pkg-postgresql/

- Debian Packaging for Extensions is on alioth SVN
  https://alioth.debian.org/scm/browser.php?group_id=30209

- Our own build infrastructure is on github
  https://github.com/dimitri/apt.postgresql.org

## Jenkins

- we want the configuration in git if possible: SCM Sync configuration plugin

-----------

http://www.df7cb.de/blog/2012/PostgreSQL_in_Debian_Hackathon.html

-----------

Almost a year has passed since my talk at pgconf.eu 2011 in Amsterdam on
<a
href="http://www.postgresql.eu/events/schedule/pgconfeu2011/session/166-connecting-the-debian-and-postgresql-worlds/">Connecting
theDebian and PostgreSQL worlds</a>, 
and unfortunately little has happened on that front, mostly due to my limited
spare time between family and job.
<a href="http://pgapt.debian.net/">pgapt.debian.net</a> is up and running, but
got few updates and is lagging behind on PostgreSQL releases.

Luckily, we got the project moving. <a href="http://tapoueh.org/blog/index.html">Dimitri Fontaine</a>
and <a href="http://blog.hagander.net/">Magnus Hagander</a> suggested to do a
face-to-face meeting, so we got together at my house for two days last week and
discussed ideas, repository layouts, build scripts, and whatnot to get all of
us aligned for pushing the project ahead. My
<a href="http://www.credativ.de/">employer</a> sponsored my time off work for
that. We almost finished moving the repository to postgresql.org
infrastructure, barring some questions of how to hook the repository into the
existing mirror infrastructure; this should get resolved this week.

The build server running Jenkins is still located on my laptop, but moving this
to a proper host will also happen really soon now. We are using
<a href="http://michael-prokop.at/blog/">Mika Prokop</a>'s
<a href="http://jenkins-debian-glue.org/">jenkins-debian-glue</a> scripts for
driving the package build from Jenkins. The big plus point about Jenkins is
that it makes executing jobs on different distributions and architectures in
parallel much easier than a bunch of homemade shell scripts could get us with
reasonable effort.

Here's a list of random points we discussed:

 * We decided to go for "pgdg" in version numbers and distribution names, i.e.
   packages will have version numbers like 9.1.5-1.pgdg+1, with distributions
   wheezy-pgdg, squeeze-pgdg, and so on.
 * There will be Debian-testing-style distributions called like
   wheezy-pgdg-testing that packages go into for some time before they get
   promoted to the "live" distributions.
 * PostgreSQL versions out of support (8.2 and below) will not be removed from
   the repository, but will be moved to distributions called like
   wheezy-pgdg-deprecated. People will still be able to use them, but the
   naming should make it clear that they should really be upgrading.
 * We have a slightly modified (compared to Debian unstable) postgresql-common
   package that sets the "supported-versions" to all versions supported by the
   PostgreSQL project. That will make the postgresql-server-dev-all package
   pull in build-dependencies for all server versions, and make extension
   module packages compile for all of them automatically. (Provided they are
   using pg_buildext.)
 * There's no Ubuntu support in there yet, but that's mostly only a matter of
   adding more cowbuilder chroots to the build jobs. TBD soon.

We really aim at using unmodified packages from Debian as much as possible, and
in fact this project doesn't mean to replace Debian's PostgreSQL packaging
work, but to extend it beyond the number of server versions (and Debian and
Ubuntu versions covered) supported. The people behind the Debian and Ubuntu
packages, and this repository are mostly the same, so we will claim that "our"
packages will be the same quality as the "original" ones. Big thanks go to
<a href="http://www.piware.de/">Martin Pitt</a> for maintaining the
postgresql-common testsuite that really covers every aspect of running
PostgreSQL servers on Debian/Ubuntu systems.

Stay tuned for updates! :)

[[!tag debian postgresql]]

-----------

There is code for webpages using the postgresql.org template in there,
but I haven't updated any content yet:
https://github.com/dimitri/apt.postgresql.org/tree/master/web
Some of it is already visible at pgapt.debian.net, but that needs more
work. (And that's the old repo location I'm only syncing occasionally
from the new one.)

The jenkins scripts are here:
https://github.com/dimitri/apt.postgresql.org/tree/master/jenkins

generate-pgdg-source has grown the ability to put the various lib
packages from the server sources (libpq5 and friends) into the right
component in the archive, depending on the distribution targeted.

There is a new "pgdg-buildenv" package that will speed up cowbuilder
builds a bit: https://github.com/ChristophBerg/pgdg-buildenv

I'm currently updating all postgresql-x.y packages to use a debian/
directory similar to what 9.1/9.2 have. (Maybe we can even have one
that will work for all?)

The bzr URLs have (mostly) changed to end in ...pgdg instead of the
old ...sid-pgapt.

The Jenkins build host is still running on my notebook, that's one of
the next things I'm going to tackle.

Christoph
--
cb@df7cb.de | http://www.df7cb.de/