Thread: Time to upgrade buildfarm coverage for some EOL'd OSes?

Hi,

I noticed that for NetBSD we only have one animal, and it's running
EOL'd release 7.  To give decent visibility of relevant portability
problems it'd be nice to have one of the current supported releases[1]
in there.  CC'ing owner; any interest in updating this animal to 9.x?

For FreeBSD the situation is better, we have HEAD (bleeding edge 14),
13.x, and then loach running 10.3 which is dead.  Given that 12.x and
13.x are supported[2] (well, 11.4 is just about done), perhaps it'd
make sense to cover 12.x rather than 10.x?

I don't know too much about DragonflyBSD, but I happened to be
surveying operating systems we support (by the "it's in the build farm
so we're going to keep it green" definition) in the context of some
AIO work, and I learned that they'd ripped the native AIO support out
of this one at some point, which caused me to focus on the versions.
Animal conchuela is running 4.4 (2016) while 6.0 is current[3].
Again, if we're going to have one example of a rare OS that someone
cares about, I think it'd be useful to have a current one?

For OpenBSD we have the current[4] and previous major releases
covered, so that's cool, and then there's a 5.9 system, which is long
dead and could probably be put to better use, but at least we don't
lack coverage there.

In all these cases there are more options that could be turned on, in
case someone is interested in extending what's tested.  From my notes:

FreeBSD 12:

pkg install -y gmake ccache git flex bison readline p5-IPC-Run llvm
pkgconf python3 libxslt openldap-client gettext tcl86 krb5

 --enable-cassert --enable-debug --enable-tap-tests --enable-nls
--with-gssapi --with-icu --with-tcl --with-perl --with-python
--with-pam --with-ldap --with-libxml --with-libxslt --with-lz4
--with-openssl --with-llvm --with-libs=/usr/local/lib
--with-includes=/usr/local/include CC="ccache cc" CXX="ccache c++"
CLANG="ccache clang90" LLVM_CONFIG="llvm-config90"

NetBSD 9:

pkgin -y install gmake git flex bison ccache readline
mozilla-rootcerts p5-IPC-Run llvm clang pkg-config icu lz4 libxslt tcl

--enable-cassert --enable-debug --enable-tap-tests --enable-nls
--with-gssapi --with-icu --with-tcl --with-perl --with-python
--with-pam --with-ldap --with-libxml --with-libxslt --with-lz4
--with-openssl --with-llvm --with-includes=/usr/pkg/include CC="ccache
cc" CXX="ccache c++" CLANG="ccache clang" LLVM_CONFIG="llvm-config"
PYTHON="python3.8" LDFLAGS="-R/usr/pkg/lib"

OpenBSD 6.9:

pkg_add -z ccache gmake git bison autoconf-2.69 readline screen--
p5-IPC-Run icu4c python3 libxml libxslt openldap-client--gssapi
tcl-8.6.8 gettext-tools

--enable-cassert --enable-debug --enable-tap-tests --enable-nls
--with-icu --with-tcl --with-perl --with-python --with-bsd-auth
--with-ldap --with-libxml --with-libxslt --with-lz4 --with-openssl
CC="ccache cc"

(I have never succeeded in getting our LLVM stuff running on OpenBSD;
there's something wrong with their LLVM package the way we use it and
their retpoline mitigation stuff, not investigated further.)

(Hmm, in hindsight, I don't know why we need "--with-bsd-auth" instead
of detecting it, but I don't plan to work on that...)

[1] https://www.netbsd.org/releases/
[2] https://www.freebsd.org/releases/
[3] https://www.dragonflybsd.org/
[4] https://www.openbsd.org/

Thomas Munro <thomas.munro@gmail.com> writes:
> (Hmm, in hindsight, I don't know why we need "--with-bsd-auth" instead
> of detecting it, but I don't plan to work on that...)

As far as that goes, I thought we had a policy against auto-detecting
user-visible features.  From memory, the rationale goes like "if you
want feature X you should say so, so that the build will fail if we
can't provide it".  Thus we make you say something like --with-openssl
even though it wouldn't be particularly hard to auto-detect.  Peter E.
can probably advocate more strongly for this approach.

But anyway, +1 for your main point that it might be time to move up
some buildfarm animals, unless we want to scrape up extra resources
to test both older and newer OS versions.

            regards, tom lane

On Fri, Oct 8, 2021 at 11:40 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Thomas Munro <thomas.munro@gmail.com> writes:
> > (Hmm, in hindsight, I don't know why we need "--with-bsd-auth" instead
> > of detecting it, but I don't plan to work on that...)
>
> As far as that goes, I thought we had a policy against auto-detecting
> user-visible features.  From memory, the rationale goes like "if you
> want feature X you should say so, so that the build will fail if we
> can't provide it".  Thus we make you say something like --with-openssl
> even though it wouldn't be particularly hard to auto-detect.  Peter E.
> can probably advocate more strongly for this approach.

Oh, I see.  I was thinking that operating system features were a bit
different from "external packages" (the purpose of --with according to
the autoconf docs), but that's a bit fuzzy and I see now that it's
consistent with our treatment of PAM which is very similar.

On 2021-10-08 00:15, Thomas Munro wrote:

> I noticed that for NetBSD we only have one animal, and it's running
> EOL'd release 7.  To give decent visibility of relevant portability
> problems it'd be nice to have one of the current supported releases[1]
> in there.  CC'ing owner; any interest in updating this animal to 9.x?

Yes, it's getting long in the tooth.  I will upgrade the NetBSD 7 
(sidewinder) to 9.2.


> For FreeBSD the situation is better, we have HEAD (bleeding edge 14),
> 13.x, and then loach running 10.3 which is dead.  Given that 12.x and
> 13.x are supported[2] (well, 11.4 is just about done), perhaps it'd
> make sense to cover 12.x rather than 10.x?

And I will also upgrade loach to 12.x if that's the version that is 
needed the most.


> I don't know too much about DragonflyBSD, but I happened to be
> surveying operating systems we support (by the "it's in the build farm
> so we're going to keep it green" definition) in the context of some
> AIO work, and I learned that they'd ripped the native AIO support out
> of this one at some point, which caused me to focus on the versions.
> Animal conchuela is running 4.4 (2016) while 6.0 is current[3].
> Again, if we're going to have one example of a rare OS that someone
> cares about, I think it'd be useful to have a current one?

I will upgrade conchuela to DragonFlyBSD 6.0.


> For OpenBSD we have the current[4] and previous major releases
> covered, so that's cool, and then there's a 5.9 system, which is long
> dead and could probably be put to better use, but at least we don't
> lack coverage there.

I will remove the 5.9 (curculio) and upgrade the 6.5 (morepork) to 6.9.

Would these changes be acceptable?

/Mikael

=?UTF-8?Q?Mikael_Kjellstr=c3=b6m?= <mikael.kjellstrom@mksoft.nu> writes:
> Yes, it's getting long in the tooth.  I will upgrade the NetBSD 7
> (sidewinder) to 9.2.
> And I will also upgrade loach to 12.x if that's the version that is
> needed the most.
> I will upgrade conchuela to DragonFlyBSD 6.0.
> I will remove the 5.9 (curculio) and upgrade the 6.5 (morepork) to 6.9.

+1 to all of that except retiring curculio.  That one has shown us issues
we've not seen on other animals (recent examples at [1][2]), so unless
you're feeling resource-constrained I'd vote for keeping it going.

            regards, tom lane

[1] https://www.postgresql.org/message-id/YPAdf9r5aJbDoHoq@paquier.xyz
[2] https://www.postgresql.org/message-id/CAA4eK1+uW1UGDHDz-HWMHMen76mKP7NJebOTZN4uwbyMjaYVww@mail.gmail.com

On 2021-10-08 18:12, Tom Lane wrote:
> =?UTF-8?Q?Mikael_Kjellstr=c3=b6m?= <mikael.kjellstrom@mksoft.nu> writes:
>> Yes, it's getting long in the tooth.  I will upgrade the NetBSD 7
>> (sidewinder) to 9.2.
>> And I will also upgrade loach to 12.x if that's the version that is
>> needed the most.
>> I will upgrade conchuela to DragonFlyBSD 6.0.
>> I will remove the 5.9 (curculio) and upgrade the 6.5 (morepork) to 6.9.
> 
> +1 to all of that except retiring curculio.  That one has shown us issues
> we've not seen on other animals (recent examples at [1][2]), so unless
> you're feeling resource-constrained I'd vote for keeping it going.

Sure I can keep curculio as is.  Will just upgrade morepork to OpenBSD 
6.9 then.

/Mikael

On Sat, Oct 9, 2021 at 5:55 AM Mikael Kjellström
<mikael.kjellstrom@mksoft.nu> wrote:
> On 2021-10-08 18:12, Tom Lane wrote:
> > =?UTF-8?Q?Mikael_Kjellstr=c3=b6m?= <mikael.kjellstrom@mksoft.nu> writes:
> >> Yes, it's getting long in the tooth.  I will upgrade the NetBSD 7
> >> (sidewinder) to 9.2.
> >> And I will also upgrade loach to 12.x if that's the version that is
> >> needed the most.
> >> I will upgrade conchuela to DragonFlyBSD 6.0.
> >> I will remove the 5.9 (curculio) and upgrade the 6.5 (morepork) to 6.9.
> >
> > +1 to all of that except retiring curculio.  That one has shown us issues
> > we've not seen on other animals (recent examples at [1][2]), so unless
> > you're feeling resource-constrained I'd vote for keeping it going.
>
> Sure I can keep curculio as is.  Will just upgrade morepork to OpenBSD
> 6.9 then.

Thanks very much for doing all these upgrades!

Here's a nice recording of a morepork, a kind of owl that we often
hear in the still of the night where I live.  Supposedly it's saying
'more pork!' but I don't hear that myself.

https://www.doc.govt.nz/nature/native-animals/birds/birds-a-z/morepork-ruru/

On 08.10.21 00:40, Tom Lane wrote:
> As far as that goes, I thought we had a policy against auto-detecting
> user-visible features.  From memory, the rationale goes like "if you
> want feature X you should say so, so that the build will fail if we
> can't provide it".  Thus we make you say something like --with-openssl
> even though it wouldn't be particularly hard to auto-detect.  Peter E.
> can probably advocate more strongly for this approach.

Yeah, there used to be RPMs shipped that accidentally omitted readline 
support.

On 2021-10-08 21:40, Thomas Munro wrote:

>> Sure I can keep curculio as is.  Will just upgrade morepork to OpenBSD
>> 6.9 then.
> 
> Thanks very much for doing all these upgrades!

No problem.

Current status is:

loach: Upgraded to FreeBSD 12.2
morepork: Upgraded to OpenBSD 6.9
conchuela: Upgraded to DragonFly BSD 6.0
sidewinder: Upgraded to NetBSD 9.2

curculio: Is not able to connect to https://git.postgresql.org due to 
the Let's Encrypt expired CA.

Without doing anything:

$ git clone https://git.postgresql.org
Cloning into 'git.postgresql.org'...
fatal: unable to access 'https://git.postgresql.org/': SSL certificate 
problem: certificate has expired

Modifying /etc/ssl/certs.pem by removing expired DST Root CA X3:

$ git clone https://git.postgresql.org
Cloning into 'git.postgresql.org'...
fatal: unable to access 'https://git.postgresql.org/': SSL certificate 
problem: unable to get local issuer certificate

Then I tried to download the new CA and Intermediate from:

https://letsencrypt.org/certificates/

and adding them manually to /etc/ssl/cert.pem

but no dice.  Only getting:

$ git clone https://git.postgresql.org
Cloning into 'git.postgresql.org'...
fatal: unable to access 'https://git.postgresql.org/': SSL certificate 
problem: unable to get local issuer certificate

If anybody have any tips about how to get SSL-working again, I'll gladly 
take it.

/Mikael

=?UTF-8?Q?Mikael_Kjellstr=c3=b6m?= <mikael.kjellstrom@mksoft.nu> writes:
> curculio: Is not able to connect to https://git.postgresql.org due to 
> the Let's Encrypt expired CA.

We're working on fixing things so that git.postgresql.org will
advertise a cert chain that is compatible with older OpenSSL
versions.  I thought that was supposed to happen this weekend,
but evidently it hasn't yet.  You will need an up-to-date
(less than several years old) /etc/ssl/certs.pem, but no software
mods should be needed.  I'd counsel just waiting a day or two
more before trying to resurrect curculio.

            regards, tom lane

On 2021-10-10 20:00, Tom Lane wrote:
> =?UTF-8?Q?Mikael_Kjellstr=c3=b6m?= <mikael.kjellstrom@mksoft.nu> writes:
>> curculio: Is not able to connect to https://git.postgresql.org due to
>> the Let's Encrypt expired CA.
> 
> We're working on fixing things so that git.postgresql.org will
> advertise a cert chain that is compatible with older OpenSSL
> versions.  I thought that was supposed to happen this weekend,
> but evidently it hasn't yet.  You will need an up-to-date
> (less than several years old) /etc/ssl/certs.pem, but no software
> mods should be needed.  I'd counsel just waiting a day or two
> more before trying to resurrect curculio.

OK.  Cool.  Then I will just sit back and relax.

Another thing I used the update_personality.pl and after that the name 
of my animals and compiler settings looks, hmm, how to say this, not 
entirely correct.

Example:

DragonFly BSD DragonFly BSD 6.0 gcc gcc 8.3 x86_64

also the status page seems to be broken.  It doesn't show any Flags anymore.

But maybe that is a known problem and someone is working on that?

/Mikael

On 2021-10-11 10:20, Magnus Hagander wrote:

> It was indeed supposed to, but didn't. It has now been done though, so 
> git.postgresql.org <http://git.postgresql.org> should now be compatible 
> with ancient OpenSSL.

And curculio is back to life and shows as all green on the status page.

So it's indeed working again.

/Mikael