Thread: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"

Summary: The TLS behavior of psql changed between 16.3 and 16.4 as shown below. If this is a bug, I kindly ask to fix it. If this is intended behavior, I kindly as for a link with instructions how to work around the problem.

Server:

I am running the official PostgreSQL 17.2 Docker Container (https://hub.docker.com/layers/library/postgres/17.2/images/sha256-c063081175f45f4a3a5ac03c234e060e67618ebe75b49e2a7ffb79f8357bd1e6) proxied by a TLSv1.3 proxy (official Traefik 3.2.3 Docker Container https://hub.docker.com/layers/library/traefik/v3.2.3/images/sha256-06966a9ba1747ad724a490b8f27df1434c64e8eee5d681df03c4761c9653f62c). Traefik utilizes ACME with Let's Encrypt to produce the TLS certificate.

I have neither reconfigured TLS in any other way, nor have I manually provided TLS certificates to neither client nor server.

Client:

Using the official PostgresSQL Docker Container (16.3 vs 16.4+), I am asking psql to connect to my server. While psql 16.3 and earlier versions successfully connect via the TLS proxy to the PostgreSQL server, psql 16.4 and later versions fail doing so:

root@hetzner-2:~# docker run -it postgres:16.3 psql "host=headcrashing.eu port=5432 dbname=postgres user=postgres password=... sslmode=require"
psql (16.3 (Debian 16.3-1.pgdg120+1), server 17.2 (Debian 17.2-1.pgdg120+1))
WARNING: psql major version 16, server major version 17.         Some psql features might not work.
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, compression: off)
Type "help" for help.

postgres=# \q
root@hetzner-2:~# docker run -it postgres:16.4 psql "host=headcrashing.eu port=5432 dbname=postgres user=postgres password=... sslmode=require"
psql: error: connection to server at "headcrashing.eu" (49.13.53.107), port 5432 failed: SSL error: tlsv1 alert no application protocol

Public Test Environment

Feel free to connect to my personal PostgreSQL 17 instance running at postgres.headcrashing.eu:5432 (TLS required).

With kind regards

-Markus

Markus KARG <markus@headcrashing.eu> writes:
> Summary: The TLS behavior of psql changed between 16.3 and 16.4 as shown 
> below.

Have you tried the current release (16.6)?

            regards, tom lane

Markus KARG <markus@headcrashing.eu> writes:

Summary: The TLS behavior of psql changed between 16.3 and 16.4 as shown 
below.

Have you tried the current release (16.6)?
			regards, tom lane

Yes, but psql 16.6 fails in the same way:

root@hetzner-2:~# docker run -it postgres:16.6 psql "host=headcrashing.eu port=5432 dbname=postgres user=postgres password=HeadcrashingPostgres2022 sslmode=require"
psql: error: connection to server at "headcrashing.eu" (49.13.53.107), port 5432 failed: SSL error: tlsv1 alert no application protocol

-Markus

On 25/12/2024 19:05, Markus KARG wrote:
> I am running the official PostgreSQL 17.2 Docker Container (https:// 
> hub.docker.com/layers/library/postgres/17.2/images/sha256- 
> c063081175f45f4a3a5ac03c234e060e67618ebe75b49e2a7ffb79f8357bd1e6) 
> proxied by a TLSv1.3 proxy (official Traefik 3.2.3 Docker Container 
> https://hub.docker.com/layers/library/traefik/v3.2.3/images/ 
> sha256-06966a9ba1747ad724a490b8f27df1434c64e8eee5d681df03c4761c9653f62c). Traefik utilizes ACME with Let's Encrypt to
producethe TLS certificate.
 

In v17, libpq requests the ALPN extension in the TLS handshake. Looks 
like the proxy doesn't know about the "postgresql" ALPN protocol, and 
rejects the connection.

I guess Traefik needs some configuration changes to tell it that the 
"postgresql" protocol is expected. Or code changes.

> Using the official PostgresSQL Docker Container (16.3 vs 16.4+), I am 
> asking psql to connect to my server. While psql 16.3 and earlier 
> versions successfully connect via the TLS proxy to the PostgreSQL 
> server, psql 16.4 and later versions fail doing so:
> 
> root@hetzner-2:~# docker run -it postgres:16.3 psql "host=headcrashing.eu port=5432 
> dbname=postgres user=postgres password=... sslmode=require"
> psql (16.3 (Debian 16.3-1.pgdg120+1), server 17.2 (Debian 17.2-1.pgdg120+1))
> WARNING: psql major version 16, server major version 17.
>           Some psql features might not work.
> SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, compression: off)
> Type"help" for help.
> 
> postgres=# \q
> root@hetzner-2:~# docker run -it postgres:16.4 psql "host=headcrashing.eu port=5432 
> dbname=postgres user=postgres password=... sslmode=require"
> psql: error: connection to server at"headcrashing.eu" (49.13.53.107), port 5432 failed: SSL error: tlsv1 alert no
applicationprotocol
 

There were no changes between 16.3 and 16.4 to explain this. When I test 
that with v16 client that I built from sources, I don't get that error.

The error message suggests that you're actually using libpq v17. And 
indeed I get that error when connecting with v17 client. Perhaps the 
postgres:16.4 docker image was built with v17 libpq?

-- 
Heikki Linnakangas
Neon (https://neon.tech)

> On 25/12/2024 19:05, Markus KARG wrote:
>> I am running the official PostgreSQL 17.2 Docker Container (https:// 
>> hub.docker.com/layers/library/postgres/17.2/images/sha256- 
>> c063081175f45f4a3a5ac03c234e060e67618ebe75b49e2a7ffb79f8357bd1e6) 
>> proxied by a TLSv1.3 proxy (official Traefik 3.2.3 Docker Container 
>> https://hub.docker.com/layers/library/traefik/v3.2.3/images/ 
>> sha256-06966a9ba1747ad724a490b8f27df1434c64e8eee5d681df03c4761c9653f62c). 
>> Traefik utilizes ACME with Let's Encrypt to produce the TLS certificate.
>
> In v17, libpq requests the ALPN extension in the TLS handshake. Looks 
> like the proxy doesn't know about the "postgresql" ALPN protocol, and 
> rejects the connection.
>
> I guess Traefik needs some configuration changes to tell it that the 
> "postgresql" protocol is expected. Or code changes.

Traefik does NOT REJECT the connection (if it would, the error message 
from psql would be different).

Traefik is "postgres-aware" already since 3.0.0, while I am running 3.2.3.

Note that psql v16.3 works fine but psql v16.4 is not, so a change 
introduced by v17 this CANNOT be the cause of the current problem.

>> Using the official PostgresSQL Docker Container (16.3 vs 16.4+), I am 
>> asking psql to connect to my server. While psql 16.3 and earlier 
>> versions successfully connect via the TLS proxy to the PostgreSQL 
>> server, psql 16.4 and later versions fail doing so:
>>
>> root@hetzner-2:~# docker run -it postgres:16.3 psql 
>> "host=headcrashing.eu port=5432 dbname=postgres user=postgres 
>> password=... sslmode=require"
>> psql (16.3 (Debian 16.3-1.pgdg120+1), server 17.2 (Debian 
>> 17.2-1.pgdg120+1))
>> WARNING: psql major version 16, server major version 17.
>>           Some psql features might not work.
>> SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, 
>> compression: off)
>> Type"help" for help.
>>
>> postgres=# \q
>> root@hetzner-2:~# docker run -it postgres:16.4 psql 
>> "host=headcrashing.eu port=5432 dbname=postgres user=postgres 
>> password=... sslmode=require"
>> psql: error: connection to server at"headcrashing.eu" (49.13.53.107), 
>> port 5432 failed: SSL error: tlsv1 alert no application protocol
>
> There were no changes between 16.3 and 16.4 to explain this. When I 
> test that with v16 client that I built from sources, I don't get that 
> error.
>
> The error message suggests that you're actually using libpq v17. And 
> indeed I get that error when connecting with v17 client. Perhaps the 
> postgres:16.4 docker image was built with v17 libpq?
I am using the original, pre-built container images found on Docker Hub 
and have NOT built them on my own. I am not a PostgresSQL committer 
either. So I cannot answer your question.

>> On 25/12/2024 19:05, Markus KARG wrote:
>>> I am running the official PostgreSQL 17.2 Docker Container (https:// 
>>> hub.docker.com/layers/library/postgres/17.2/images/sha256- 
>>> c063081175f45f4a3a5ac03c234e060e67618ebe75b49e2a7ffb79f8357bd1e6) 
>>> proxied by a TLSv1.3 proxy (official Traefik 3.2.3 Docker Container 
>>> https://hub.docker.com/layers/library/traefik/v3.2.3/images/ 
>>> sha256-06966a9ba1747ad724a490b8f27df1434c64e8eee5d681df03c4761c9653f62c). 
>>> Traefik utilizes ACME with Let's Encrypt to produce the TLS 
>>> certificate.
>>
>> In v17, libpq requests the ALPN extension in the TLS handshake. Looks 
>> like the proxy doesn't know about the "postgresql" ALPN protocol, and 
>> rejects the connection.
>>
>> I guess Traefik needs some configuration changes to tell it that the 
>> "postgresql" protocol is expected. Or code changes.

Following this theory, sslnegotiation=postgres used with psql v17.x 
should solve the problem, as it falls back to the same mechanism 
supported in v16.3, correct? Unfortunately it fails with the same error 
message:

root@hetzner-2:~# docker run -it postgres:17.2 psql 
"host=headcrashing.eu port=5432 dbname=postgres user=postgres 
password=... sslmode=require sslnegotiation=postgres"

psql: error: connection to server at "headcrashing.eu" (49.13.53.107), 
port 5432 failed: SSL error: tlsv1 alert no application protocol

The same happens when using sslnegotication=direct.

Regarding your question: I have tried "sslnegotiation=postgres" and 
"ssqlnegotiation=direct" with the original official PostgreSQL Docker 
Container found on Docker Hub. Using v16.3 psql says it doesn't know the 
option "sslnegotiation", and 16.4...17 say "tlsv1 alert no application 
protocol". Does PROOFS your claim that the original images on Docker Hub 
are using libpq v17 even for psql v16.4+!

-Markus

-Markus

Markus KARG <markus@headcrashing.eu> writes:
> Following this theory, sslnegotiation=postgres used with psql v17.x 
> should solve the problem, as it falls back to the same mechanism 
> supported in v16.3, correct?

No, sorry, it looks like we insert the ALPN extension into the SSL
request packet regardless of that.  Maybe there should have been
a way to suppress that, but v17 libpq doesn't provide one.

> Regarding your question: I have tried "sslnegotiation=postgres" and 
> "ssqlnegotiation=direct" with the original official PostgreSQL Docker 
> Container found on Docker Hub.

Let's clarify something here: there is nothing "official" about either
that docker container or anything else you might find on DockerHub.
The Postgres community does not produce any such packaging.  You'd
have to discussion the inclusion of v17 libpq with whoever did build
that container.  I know that Debian thinks it's a good idea to use
latest libpq with older servers, and this builder might be following
their lead.  (The timing would be about right, since v17.0 came out
about the same time as 16.4.)  This example does show that there
are pitfalls in that policy.

            regards, tom lane

Markus KARG <markus@headcrashing.eu> writes:

Following this theory, sslnegotiation=postgres used with psql v17.x 
should solve the problem, as it falls back to the same mechanism 
supported in v16.3, correct?

No, sorry, it looks like we insert the ALPN extension into the SSL
request packet regardless of that.  Maybe there should have been
a way to suppress that, but v17 libpq doesn't provide one.

Thank you, Tom, you saved my day! I have explicitly added the name `postgresql` to the ALPN configuration on Traefik, and immediately psql v16.4+ (including psql v17) worked like a charm!

As a reference for others, in `traefik-dynamic.yaml` add the follwing to make it work (as long as you patch the `default` TLS options, take care to keep all those protocols contained; alternative path a per-router tls config instead to ONLY have postgresql in that list):

Regarding your question: I have tried "sslnegotiation=postgres" and 
"ssqlnegotiation=direct" with the original official PostgreSQL Docker 
Container found on Docker Hub.

Let's clarify something here: there is nothing "official" about either
that docker container or anything else you might find on DockerHub.
The Postgres community does not produce any such packaging.  You'd
have to discussion the inclusion of v17 libpq with whoever did build
that container.  I know that Debian thinks it's a good idea to use
latest libpq with older servers, and this builder might be following
their lead.  (The timing would be about right, since v17.0 came out
about the same time as 16.4.)  This example does show that there
are pitfalls in that policy.

Thanks a lot for this really essential information! In fact, the authors of that official docker image (they really call it that way) call themselves "The PostgreSQL Docker Community". This is totally misleading, as one understands that to be "The Docker Community withing the PostgreSQL Community", which it (apparently) is not. Instead, it seems that it actually is "The PostgreSQL Community at Docker, Inc.", people totally unrelated to the "real" PostgreSQL Community!

Thanks a lot, Tom, for your kind help. You provided the best Christmas present this year! :-)