Thread: apt.postgresql.org repo via https will fail will some users starting 2021-10-01

Hello,

sending this here as looks like https://apt.postgresql.org is affected by this so this could trigger some support/user
questions.

Note this only (!) happens when using https:// in sources.list for the pgdg repo.

Benefit of that is debatable (see recent debian-devel discussion) but i would not be surprised if some/many people use
it.

Trigger:
https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816

End of this month some CA cert will expire related to Let's Encrypt which will trigger an bug in clients using old
openssl/gnutls.

apt is using gnutls backend and at least the version in Ubuntu <= 18.04 are affected and "apt update" will already fail
forpeople starting that date.
 

Note that canonical is working in patching gnutls so if that finishes in time and (!) if people update before that date
allgood.
 

If not they will get error similar to:
Err:9 https://apt.postgresql.org/pub/repos/apt focal-pgdg Release               
  Certificate verification failed: The certificate is NOT trusted. The revocation or OCSP data are old and have been
superseded.The certificate chain uses expired certificate.  Could not handshake: Error in the certificate verification.
[IP:87.238.57.227 443]
 

Can be triggered today i.e. with:

faketime "2021-10-01" apt update

Ideas:
- Do nothing apt.postgresql suggest http:// in the instructions
- Some on the website
- Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this bug but
breakingcompatibility with old Android
 

- Raise as bug to debian also (against openssl/gnutls) to maybe patch both in stable also to avoid this ?
  - Not sure if that is a interesting/acceptable material for stable/old-stable?

Please CC me on replies i'm not on the list

Stefan

Re: Stefan Huehner
> sending this here as looks like https://apt.postgresql.org is affected by this so this could trigger some
support/userquestions.
 
> 
> Note this only (!) happens when using https:// in sources.list for the pgdg repo.

Hi,

thanks for sharing this.

We aren't advertising https:// for apt.postgresql.org anywhere, but
the download instructions tell users to "wget" the repository key from
https://www.postgresql.org, so we are at least somewhat affected.
(wget is using gnutls at least in unstable.)

> Ideas:
> - Do nothing apt.postgresql suggest http:// in the instructions
> - Some on the website
> - Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this bug
butbreaking compatibility with old Android
 

That's probably rather the ca-certificates package?

> - Raise as bug to debian also (against openssl/gnutls) to maybe patch both in stable also to avoid this ?
>   - Not sure if that is a interesting/acceptable material for stable/old-stable?

If stretch/buster/bullseye are affected, these should be fixed, yes.

Though none of this is material for the PostgreSQL packages, can you
raise the issue with the LTS team?

Christoph

On Thu, Sep 09, 2021 at 02:33:49PM +0200, Christoph Berg wrote:
> Re: Stefan Huehner
> > sending this here as looks like https://apt.postgresql.org is affected by this so this could trigger some
support/userquestions.
 
> > 
> > Note this only (!) happens when using https:// in sources.list for the pgdg repo.
> 
> Hi,
> 
> thanks for sharing this.
> 
> We aren't advertising https:// for apt.postgresql.org anywhere, but
> the download instructions tell users to "wget" the repository key from
> https://www.postgresql.org, so we are at least somewhat affected.
> (wget is using gnutls at least in unstable.)
> 
> > Ideas:
> > - Do nothing apt.postgresql suggest http:// in the instructions
> > - Some on the website
> > - Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this bug
butbreaking compatibility with old Android
 
> 
> That's probably rather the ca-certificates package?

Not in this case, i know a bit confusing.
That upstream article has more details:
https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816
Part: How to support older OpenSSL versions

In (not so) short: ca-certificates is fine to have trust anchor for Lets Encrypt.
However not everybody directly trust Let's Encrypt (missing entry in their equivalent of ca-certificates (i.e. old
Android).

To keep those other clients supported they employed a bit of a trick which has an 'expired root certificates' in the
chainfrom your server-cert to their root. At the same time there is 2nd valid path. But old version of software
(openssl,gnutls)just stop + fail on seeing 'expired'.
 

Best they could do if offer server owner (certbot parameter when requesting ssl certificate to select):
a.) Default chain (compatible still with old android) but triggering this bug
b.) Alternative chain (ignore old android) but keep compatible with old openssl/gnutls

That link goes into much more detail but hopefully now clearer.

That is also why i raised this here as a choice for apt.postgresql.org hosting (if you think it's a useful workaround)

> 
> > - Raise as bug to debian also (against openssl/gnutls) to maybe patch both in stable also to avoid this ?
> >   - Not sure if that is a interesting/acceptable material for stable/old-stable?
> 
> If stretch/buster/bullseye are affected, these should be fixed, yes.
> 
> Though none of this is material for the PostgreSQL packages, can you
> raise the issue with the LTS team?

Will raise there.

Hopefuly above also clarified why i sent that here (not about any PostgreSQL package, but apt.postgresql.org server
admintopic).
 

Stefan

Re: Stefan Huehner
> > > - Some on the website
> > > - Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this
bugbut breaking compatibility with old Android
 
> > 
> > That's probably rather the ca-certificates package?
> 
> Not in this case, i know a bit confusing.
> That upstream article has more details:
> https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816
> Part: How to support older OpenSSL versions
> 
> In (not so) short: ca-certificates is fine to have trust anchor for Lets Encrypt.
> However not everybody directly trust Let's Encrypt (missing entry in their equivalent of ca-certificates (i.e. old
Android).
> 
> To keep those other clients supported they employed a bit of a trick which has an 'expired root certificates' in the
chainfrom your server-cert to their root. At the same time there is 2nd valid path. But old version of software
(openssl,gnutls)just stop + fail on seeing 'expired'.
 
> 
> Best they could do if offer server owner (certbot parameter when requesting ssl certificate to select):

Ah, I thought you meant the end-users servers running PostgreSQL when
you said "server".

For changing the webservers, we'd need to get pginfra on board, Cc'ed
now.

Christoph

On Thu, Sep 09, 2021 at 05:33:51PM +0200, Christoph Berg wrote:
> Re: Stefan Huehner
> > > > - Some on the website
> > > > - Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this
bugbut breaking compatibility with old Android
 
> > > 
> > > That's probably rather the ca-certificates package?
> > 
> > Not in this case, i know a bit confusing.
> > That upstream article has more details:
> > https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816
> > Part: How to support older OpenSSL versions
> > 
> > In (not so) short: ca-certificates is fine to have trust anchor for Lets Encrypt.
> > However not everybody directly trust Let's Encrypt (missing entry in their equivalent of ca-certificates (i.e. old
Android).
> > 
> > To keep those other clients supported they employed a bit of a trick which has an 'expired root certificates' in
thechain from your server-cert to their root. At the same time there is 2nd valid path. But old version of software
(openssl,gnutls)just stop + fail on seeing 'expired'.
 
> > 
> > Best they could do if offer server owner (certbot parameter when requesting ssl certificate to select):
> 
> Ah, I thought you meant the end-users servers running PostgreSQL when
> you said "server".
Sorry for the confusion.

But now thinking they could be affected (but special cases only)
- There may be a case for running PostgreSQL instances
- As this affects any 'client' using above older libraries
- libpq linked with old ssl, connecting via SSL to remote pg having Let's Encrypt and client validating certificate
(verify-ca,verify-full)
 
- or outgoing connections using fdw 

Note:
This is just me trying to construct a flow which might fail.
Again kind of 'info for supporting users' as bug is libssl we link against only.
Maybe also interesting to spread to the yum-side of things (Devrim?) to check thei rpm using packaging/distros.
But also not sure if that is common enough even to make a big topic out of...

> 
> For changing the webservers, we'd need to get pginfra on board, Cc'ed
> now.

For the sysadmins:
- 'changing' would avoid the bug described here + not for debian
  https://lists.debian.org/debian-lts/2021/09/msg00008.html
  for people running old distros (no fix, or not updated, ...)
- But it will break using the webserver with other clients (i.e. older android)
- Need to pick the 'smaller problem' based on the concrete site and its users/clients
- This is generic Let's Encrypt topic (not just apt.postgresql.org host)
- Above letsencrypt link has bigger explanation + all the background

Stefan

Re: Stefan Huehner
> But now thinking they could be affected (but special cases only)
> - There may be a case for running PostgreSQL instances
> - As this affects any 'client' using above older libraries

On stretch and later, PG is linked against libssl1.1 which is safe, so
we are not affected. (jessie is EOL.)

Christoph

On Wed, Sep 8, 2021 at 6:42 PM Stefan Huehner <stefan@huehner.org> wrote:
>
> Hello,
>
> sending this here as looks like https://apt.postgresql.org is affected by this so this could trigger some
support/userquestions. 
>
> Note this only (!) happens when using https:// in sources.list for the pgdg repo.
>
> Benefit of that is debatable (see recent debian-devel discussion) but i would not be surprised if some/many people
useit. 
>
> Trigger:
> https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816
>
> End of this month some CA cert will expire related to Let's Encrypt which will trigger an bug in clients using old
openssl/gnutls.
>
> apt is using gnutls backend and at least the version in Ubuntu <= 18.04 are affected and "apt update" will already
failfor people starting that date. 
>
> Note that canonical is working in patching gnutls so if that finishes in time and (!) if people update before that
dateall good. 
>
> If not they will get error similar to:
> Err:9 https://apt.postgresql.org/pub/repos/apt focal-pgdg Release
>   Certificate verification failed: The certificate is NOT trusted. The revocation or OCSP data are old and have been
superseded.The certificate chain uses expired certificate.  Could not handshake: Error in the certificate verification.
[IP:87.238.57.227 443] 
>
> Can be triggered today i.e. with:
>
> faketime "2021-10-01" apt update
>
> Ideas:
> - Do nothing apt.postgresql suggest http:// in the instructions
> - Some on the website
> - Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this bug
butbreaking compatibility with old Android 
>
> - Raise as bug to debian also (against openssl/gnutls) to maybe patch both in stable also to avoid this ?
>   - Not sure if that is a interesting/acceptable material for stable/old-stable?

Hi!

We've started looking into what can and should be done on the infra
side to see if we can get this working.

One question though. In my attempts to reproduce, it seems that *wget*
on Ubuntu 18.04 has no problem with the current chain, just apt-get,
does that match with your testing?  So if one follows our instructions
of getting the gpg key with https but the actual repo with http, it
never actually presents a problem?

That's not saying we don't need to do anything about it, just to
reconfirm our tests. For example, this appears to also break RedHat 6
as well...

--
 Magnus Hagander
 Me: https://www.hagander.net/
 Work: https://www.redpill-linpro.com/

On Tue, Sep 14, 2021 at 04:33:01PM +0200, Magnus Hagander wrote:
> On Wed, Sep 8, 2021 at 6:42 PM Stefan Huehner <stefan@huehner.org> wrote:
> >
> > Hello,
> >
> > sending this here as looks like https://apt.postgresql.org is affected by this so this could trigger some
support/userquestions.
 
> >
> > Note this only (!) happens when using https:// in sources.list for the pgdg repo.
> >
> > Benefit of that is debatable (see recent debian-devel discussion) but i would not be surprised if some/many people
useit.
 
> >
> > Trigger:
> > https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816
> >
> > End of this month some CA cert will expire related to Let's Encrypt which will trigger an bug in clients using old
openssl/gnutls.
> >
> > apt is using gnutls backend and at least the version in Ubuntu <= 18.04 are affected and "apt update" will already
failfor people starting that date.
 
> >
> > Note that canonical is working in patching gnutls so if that finishes in time and (!) if people update before that
dateall good.
 
> >
> > If not they will get error similar to:
> > Err:9 https://apt.postgresql.org/pub/repos/apt focal-pgdg Release
> >   Certificate verification failed: The certificate is NOT trusted. The revocation or OCSP data are old and have
beensuperseded. The certificate chain uses expired certificate.  Could not handshake: Error in the certificate
verification.[IP: 87.238.57.227 443]
 
> >
> > Can be triggered today i.e. with:
> >
> > faketime "2021-10-01" apt update
> >
> > Ideas:
> > - Do nothing apt.postgresql suggest http:// in the instructions
> > - Some on the website
> > - Think on reconfiguring certbot/Let's Encrypt on the server to switch to the alternative chain (avoiding this bug
butbreaking compatibility with old Android
 
> >
> > - Raise as bug to debian also (against openssl/gnutls) to maybe patch both in stable also to avoid this ?
> >   - Not sure if that is a interesting/acceptable material for stable/old-stable?
> 
> Hi!
> 
> We've started looking into what can and should be done on the infra
> side to see if we can get this working.
> 
> One question though. In my attempts to reproduce, it seems that *wget*
> on Ubuntu 18.04 has no problem with the current chain, just apt-get,
> does that match with your testing?  So if one follows our instructions
> of getting the gpg key with https but the actual repo with http, it
> never actually presents a problem?

Hi Magnus,

sorry for the delay i was out travelling.

I just checked and and wget for Ubuntu 18.04/bionic is linked to libssl1.1 not having the problem in the first place.
Which explains why is it not affected here. I think wget can have both gnutls+openssl backend so need to see which is
usedper distro/release (As can change)
 

# apt show wget | grep Depends
Depends: libc6 (>= 2.17), libidn2-0 (>= 0.6), libpcre3, libpsl5 (>= 0.16.0), libssl1.1 (>= 1.1.0), libuuid1 (>= 2.16)

Also for apt itself in 18.04 Ubuntu has backported the fix into gnutls30 and it just reached the normal update
repository.

Package/version containing the fix is:
libgnutsl30    3.5.18-1ubuntu1.5

So here if people install (non-security) updates they will be not affected.
Or telling them "run system updates" if they get the bug is enough.

Note for older Ubuntu 16.04+14.04 Ubuntu offers paid updates still which are not yet done /published for this issue but
alsoin progress.
 

For debian also fixes are moving along after i reached our to debian-lts list ass Christoph suggested earlier.

> That's not saying we don't need to do anything about it, just to
> reconfirm our tests. For example, this appears to also break RedHat 6
> as well...

But good news is also impact is getting smaller with less distro/releases affected as some are getting fixed.

Also with few combinations being affected maybe just some extra note/section in the wiki could be enough.
wget http:// for affected + extra step to verify key fingerprint

For other distros (RHEL,Centos,Suse, ...) do we have anyone having contact's there nearby here?
I think there was yum.postgresql.org with Devrim active on that (but don't remember).
As probably reaching out to them maybe they are also interested in just backporting the libary fixes on their side.

Stefan

> 
> -- 
>  Magnus Hagander
>  Me: https://www.hagander.net/
>  Work: https://www.redpill-linpro.com/