Re: Hackathon news - Mailing list pgsql-pkg-debian

Re: Martin Pitt 2012-09-07 <20120907063015.GD2625@piware.de>
> >  1. get our own postgresql-common package with the right supported-versions script
>
> We already discussed this briefly on IRC. This seems to be the only
> delta required, which is both tiny as well as bothersome as it won't
> ever go away. The difference between merging to new p-common releases
> and just pulling them unmodified into pgapt is considerable, so I
> think we should find a way to avoid this permanent delta.
>
> As you already plan to have a separate package which provides the apt
> pinning, could we not just use the same package to divert
> postgresql-client-common's
> /usr/share/postgresql-common/supported-versions script ? Or we change
> supported-versions to check if a file or script
> /usr/share/postgresql-common/supported-versions.local exists and
> return its contents/run it and return its output?

Afaict, there are two places where supported-versions really matters,
that is at p-common build time (to determine which PG version
postgresql.deb should depend on), and extension build time (to
determine which PG versions to build extensions for). The case where
the user is notified that he's running an out-of-date versions is not
so important. I'm not aware of other situations where it is used.

This seems to imply we could just make sure our build environments
have the right s-v script, and maybe even ship an unmodified version
to users, or go via some .local variant.

We need a "pgdg-repository" package on user systems for several
reasons:
* ship the repository key (which I need to refresh soonish, it's going
  to expire in October)
* possibly auto-add some /etc/apt/sources.list.d/pgdg.list file
* add some /etc/apt/preferences.d/pgdg file (these two should probably
  be debconf questions)
* Do the s-v tweaks mentioned above.

I'll try to build some poc for that later.

> > ### distributions and release policy
> >
> > We publish maintained PostgreSQL versions to current debian and ubuntu
> > releases. We also publish old PostgreSQL versions to current and old debian
> > and ubuntu versions.
>
> Do you plan to include running the p-common test suite in Jenkins, to
> ensure that there are no mis-builds or other things we haven't thought
> of (e. g. a p-common operation that does not work with 8.3)?

That's already up and running, the build pipeline in Jenkins is:
* the postgresql-x.y-source job produces postgresql_x.y-pgdgZZ+1.dsc
  where ZZ depends on the distribution (generate-pgdg-source)
* the postgreqsql-x.y-binaries job compiles the sources for all
  distributions and architectures (build-and-provide-package)
* then a common postgresql-testsuite job grabs the binaries from the
  last build and runs the testsuite for every distribution and
  architecture (postgresql-testsuite)
* finally a common dput job uploads everything to the various
  *-pgdg-testing distributions in the archive. (dput-pgdg)

If any of the first three fails, there will be no upload.

The scripts mentioned in parentheses are available at
https://github.com/dimitri/apt.postgresql.org/tree/master/jenkins
https://github.com/mika/jenkins-debian-glue/tree/master/scripts

The Jenkins setup will hopefully be publically visible soonish. I'll
also write some more documentation on how the jobs need to be
configured.

The -source script does essentially only add a changelog entry to get
the version numbers right. For the server packages (it is also used
for extension and general packages), it will also move the lib
packages (libpq5, libpq-dev, ...) to some other component in the
archive, except for the current stable PG version (beta/rc in
unstable).

IOW, this means:

postgresql-9.0's libpq will go to Section: 9.0/libs.
postgresql-9.1's libpq will go to Section: libs (this is implicitely
main/libs) for squeeze and wheezy, and to 9.1/libs for unstable
postgresql-9.2's libpq will to to Section 9.2/libs for squeeze and
wheezy, and to libs for unstable.

That way, the most recent relevant libpq is always available in a
distribution, but users can still install the others if they want to,
then just need to change:

deb http://... main

to

deb http://... main 8.3 8.4 9.2

This tweak was the reason why I put all those "sid-pgapt" branches on
bzr.debian.org, but now this tweak is done on the fly. (Which looks
surprisingly similar to the last patch Martin did on this ppa script.
I didn't look there before, though :)

As said in the last mail, the branches will be renamed to "pgdg" where
they are still useful (9.0 and below, I don't want to change the trunk
branch from the last version found in Debian), the others (9.1 and
9.2) will go away soon.

> > Several sources are available for the packaging. We can either build from
> > the current `apt-get source` packaging found in sid or from the source code
> > repository on `alioth` (svn and git are in use over there).
>
> The Debian PostgreSQL packages are in bzr. But I'd advise to only
> build from released versions. To avoid having to wait for the Debian
> archive processing, how about building packages from tags only,
> instead of from arbitrary revisions? Then jenkins could watch out for
> new tags, and immediately get going when it sees one, instead of
> having to wait for half a day for the Debian archive publisher.

That's the plan, yes. Unfortunately, the Jenkins bzr plugin doesn't
support that, but as builds are triggered manually at the moment, that
will work too.

We also experimented a bit with using "apt-get source" instead of
pulling from some VCS. It should work equally well, except that this
will need some waiting as you said.

> > One idea is to build *testing* packages from their source code repositories
> > and *production* package directely from the `sid` distribution. That means
> > we're doing both Quality Assurance and Backports, but that might be a little
> > too much for the first version of this build system.
>
> The testing one is interesting if we could combine it with p-common
> test suite runs (but that should happen for production packages as
> well). But yes, it's a matter of extra hardware resources.

We were discussing this for quite some time, but the issue isn't
really clear from the notes. The question was: do we pull extension
packages from unstable, and put them in all our distributions, or do
we pull the version from the corresponding Debian distribution? Even
if we were only rebuilding the versions Debian already had way, we
would still provide users more PG versions targetted. For example:

Debian currently has:

sid: plproxy 2.4-1 with binary plproxy-9.1
squeeze plproxy 2.1-1 with binary plproxy-8.4

Variant #1 would be

sid-pgdg plproxy 2.4-1.pgdg+1 with binaries plproxy-{8.3,8.4,9.0,9.1,9.2}
squeeze-pgdg plproxy 2.4-1.pgdg60+1 with binaries plproxy-{8.3,8.4,9.0,9.1}
                       ^

Variant #2 would be

sid-pgdg plproxy 2.4-1.pgdg+1 with binaries plproxy-{8.3,8.4,9.0,9.1,9.2}
squeeze-pgdg plproxy 2.1-1.pgdg60+1 with binaries plproxy-{8.3,8.4,9.0,9.1}
                       ^

Both have their advantages, and we could even do both at the same
time. We then decided that this would be insanely complex to handle
(and to use), and we would do #1 only.

> > ## publishing policy
> >
> > Generally we want to only include packages that come from Debian unstable,
> > usually in the version from there. Exceptions should be rare so we don't run
> > out of sync and lose track.
>
> Don't you also want to track experimental? That's where I put all the
> betas and RCs, as well as 9.2 right now while we are in release
> freeze.

The emphasis was on "from Debian". Experimental is of course also a
good place to take things from, especially during the freeze.

> Thanks!

Thanks for the feedback!

Christoph
--
cb@df7cb.de | http://www.df7cb.de/