Thread: BUG #12769: SSL-Renegotiation failures

BUG #12769: SSL-Renegotiation failures

From
pilum.70@uni-muenster.de
Date:
The following bug has been logged on the website:

Bug reference:      12769
Logged by:          Arne Scheffer
Email address:      pilum.70@uni-muenster.de
PostgreSQL version: 9.3.6
Operating system:   CentOS6, openssl version OpenSSL 1.0.1e-fips
Description:

I get ssl renegotiation failures with streaming standbys. Sometimes the
connection breaks and is reconnected afterwards. However, if I use
pg_basebackup (same libpq connection string), I don't get any of these
failures, allthough the transferred data ist far beyond 512 MB
So I don't think it's the
ssl renegotiation bug (openssl of a yum update patched centos6)
If I disable ssl_renegotiation_limit to 0, there are no errors any more,
but that is only a workaround, no solution.

Steps to reproduce with testdata:

Tested on patched centos6, Postgresql 9.3.6

bash-4.1$ openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013

Install Repository

yum install postgresql93 postgresql93-server postgresql93-contrib
postgresql93-devel postgresql93-libs

/usr/pgsql-9.3/bin/initdb -D /tmp/data1

Not necessary but comfortable: create two postgresql-93test Init-Scripts (cp
/etc/init.d/postgresql-9.3 postgresql-93test)
- changing data-Directory to /tmp/data1 und /tmp/data2
- changing PGPORT to yourport and yourport2

Start server in data1
On server in data1:
create user replicateme with replication encrypted password "insecure";

<#####
postgresql.conf:
ssl = on
listen_addresses = 'yourserver.domain.com'
wal_level = hot_standby
# 4GB fuer WAL-Segmente:
wal_keep_segments = 256
max_wal_senders = 2
#####>

Certificates installed

<##### only 1 line:
pg_hba.conf:
hostssl replication ...
#####>

usr/pgsql-9.3/bin/pg_basebackup -D /tmp/data2 -v -P -Fp -l testsslbackup -x
-d "host=yourserver.domain.com port=yourport user=replicator
sslmode=verify-full"

<####
recovery.conf:

recovery_target_timeline = 'latest'
standby_mode = 'on'
primary_conninfo = 'host=yourserver.domain.com port=yourport user=replicator
sslmode=verify-full'
#####>

Start standby server in data2

On server in data1:

create database test
\c test
create table test2 (a int, b text);
insert into test2 (select generate_series(0,5000000,1));
update test2 set
b='sljfdlksjaofjsaolfjsdlkofjdolasjfoldsjfoldsjaolfjdloksjfolajsdflkjsdlokfjslojdalkfjoldsjfoslafjolasljfdloas';

tail -f /tmp/data1/pg_log/postgresql-Fri.log
[...]
< 2015-02-13 18:09:55.964 CET >LOG:  SSL renegotiation failure
[...]
< 2015-02-13 18:10:11.819 CET >LOG:  SSL renegotiation failure
[...]


usr/pgsql-9.3/bin/pg_basebackup -D /tmp/data3 -v -P -Fp -l testsslbackup2 -x
-d "host=yourserver.domain.com port=yourport user=replicator
sslmode=verify-full"

tail -f /tmp/data1/pg_log/postgresql-Fri.log
--> nothing!

Re: BUG #12769: SSL-Renegotiation failures

From
Andres Freund
Date:
Hi Arne,

On 2015-02-13 18:52:02 +0000, pilum.70@uni-muenster.de wrote:
> The following bug has been logged on the website:
>
> Bug reference:      12769
> Logged by:          Arne Scheffer
> Email address:      pilum.70@uni-muenster.de
> PostgreSQL version: 9.3.6
> Operating system:   CentOS6, openssl version OpenSSL 1.0.1e-fips
> Description:
>
> I get ssl renegotiation failures with streaming standbys. Sometimes the
> connection breaks and is reconnected afterwards. However, if I use
> pg_basebackup (same libpq connection string), I don't get any of these
> failures, allthough the transferred data ist far beyond 512 MB
> So I don't think it's the
> ssl renegotiation bug (openssl of a yum update patched centos6)
> If I disable ssl_renegotiation_limit to 0, there are no errors any more,
> but that is only a workaround, no solution.

Heikki and me have recently investigated problems around SSL
renegotiation. See
http://www.postgresql.org/message-id/20150126101405.GA31719@awork2.anarazel.de
.

Greetings,

Andres Freund

Re: BUG #12769: SSL-Renegotiation failures

From
Heikki Linnakangas
Date:
On 02/13/2015 10:59 PM, Andres Freund wrote:
> On 2015-02-13 18:52:02 +0000, pilum.70@uni-muenster.de wrote:
>> I get ssl renegotiation failures with streaming standbys. Sometimes the
>> connection breaks and is reconnected afterwards. However, if I use
>> pg_basebackup (same libpq connection string), I don't get any of these
>> failures, allthough the transferred data ist far beyond 512 MB
>> So I don't think it's the
>> ssl renegotiation bug (openssl of a yum update patched centos6)
>> If I disable ssl_renegotiation_limit to 0, there are no errors any more,
>> but that is only a workaround, no solution.
>
> Heikki and me have recently investigated problems around SSL
> renegotiation. See
> http://www.postgresql.org/message-id/20150126101405.GA31719@awork2.anarazel.de
> .

I wasn't able to reproduce exactly the same error you saw, Arne, so it
would be good if you could test the patches I've been developing, to see
if they fix your problem too. That is, patches 0001 and 0002 from
http://www.postgresql.org/message-id/54DE6FAF.6050005@vmware.com. Could
you do that?

- Heikki

Re: BUG #12769: SSL-Renegotiation failures

From
Arne Scheffer
Date:
On the same machine (CentOS6)

Cloned from 9.5devel,
./configure --with-perl --with-openssl --with-python --with-tcl --with-pam --with-ldap --enable-thread-safety
--enable-debug
make make install make clean

Repeated the procedure attached in the mail
Got a different error (also twice at exected renegotiation times):

< 2015-02-15 16:40:45.438 CET >LOG:  SSL error: session id context uninitialized
< 2015-02-15 16:40:45.439 CET >LOG:  could not receive data from client: Connection reset by peer
< 2015-02-15 16:40:45.439 CET >LOG:  unexpected EOF on standby connection

Tried

git checkout -b ssl_patch
patch -p1 <../0001-Fix-sslv3-alert-unexpected-message-errors-in-SSL-ren.patch
(got applied on 2 files)
patch -p1 <../0002-Also-drain-input-buffer-in-non-blocking-mode-if-send.patch
(got applied on 1 file)

Repeated make make install make clean

Repeated the procedure attached in the mail.

(Both twice.)

Got the same errors.
Perhaps I did something wrong. Could you add a temporary debug line, so that I can
see, that the patch is really applied in my environment?

Even tried 0003, but no change.

Patch expectedly doesn't apply on 9.3.6:

[root@zivwebapp13 postgresql-9.3.6patched]# patch -p1
<../0001-Fix-sslv3-alert-unexpected-message-errors-in-SSL-ren.patch
patching file src/interfaces/libpq/fe-misc.c
Hunk #1 succeeded at 919 (offset -1 lines).
can't find file to patch at input line 45
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/src/interfaces/libpq/fe-secure-openssl.c
b/src/interfaces/libpq/fe-secure-openssl.c
|index a32af34..93b8184 100644
|--- a/src/interfaces/libpq/fe-secure-openssl.c
|+++ b/src/interfaces/libpq/fe-secure-openssl.c
--------------------------

I would also test backpatched patch code once it's made.

VlG

Arne


On Sat, 14 Feb 2015, Heikki Linnakangas wrote:

> On 02/13/2015 10:59 PM, Andres Freund wrote:
>> On 2015-02-13 18:52:02 +0000, pilum.70@uni-muenster.de wrote:
>>> I get ssl renegotiation failures with streaming standbys. Sometimes the
>>> connection breaks and is reconnected afterwards. However, if I use
>>> pg_basebackup (same libpq connection string), I don't get any of these
>>> failures, allthough the transferred data ist far beyond 512 MB
>>> So I don't think it's the
>>> ssl renegotiation bug (openssl of a yum update patched centos6)
>>> If I disable ssl_renegotiation_limit to 0, there are no errors any more,
>>> but that is only a workaround, no solution.
>>
>> Heikki and me have recently investigated problems around SSL
>> renegotiation. See
>> http://www.postgresql.org/message-id/20150126101405.GA31719@awork2.anarazel.de
>> .
>
> I wasn't able to reproduce exactly the same error you saw, Arne, so it would
> be good if you could test the patches I've been developing, to see if they
> fix your problem too. That is, patches 0001 and 0002 from
> http://www.postgresql.org/message-id/54DE6FAF.6050005@vmware.com. Could you
> do that?
>
> - Heikki
>

Re: BUG #12769: SSL-Renegotiation failures

From
Arne Scheffer
Date:
Retried my test suite after commit

1c2b7c0879d83ff79e4adf2c0a883df92b713da4     Restore the SSL_set_session_id_context() call to OpenSS...

Cloned again from 9.5devel master.
Now the two patches work as expected (Means: errors without patch, no errors
with patch (I can't see, whether the code is really performing renegotiation)).
That seems promising.

As already mentioned:
I would also test the two patches for 9.3
in a production near environment to prove them fixing the reported Bug.
If that isn't desired,
I will clone them from REL9_3_STABLE branch once they are committed.

VlG-Arne


On Sun, 15 Feb 2015, Arne Scheffer wrote:

> On the same machine (CentOS6)
>
> Cloned from 9.5devel,
> ./configure --with-perl --with-openssl --with-python --with-tcl --with-pam
> --with-ldap --enable-thread-safety --enable-debug
> make make install make clean
>
> Repeated the procedure attached in the mail
> Got a different error (also twice at exected renegotiation times):
>
> < 2015-02-15 16:40:45.438 CET >LOG:  SSL error: session id context
> uninitialized
> < 2015-02-15 16:40:45.439 CET >LOG:  could not receive data from client:
> Connection reset by peer
> < 2015-02-15 16:40:45.439 CET >LOG:  unexpected EOF on standby connection
>
> Tried
>
> git checkout -b ssl_patch
> patch -p1 <../0001-Fix-sslv3-alert-unexpected-message-errors-in-SSL-ren.patch
> (got applied on 2 files)
> patch -p1 <../0002-Also-drain-input-buffer-in-non-blocking-mode-if-send.patch
> (got applied on 1 file)
>
> Repeated make make install make clean
>
> Repeated the procedure attached in the mail.
>
> (Both twice.)
>
> Got the same errors.
> Perhaps I did something wrong. Could you add a temporary debug line, so that
> I can
> see, that the patch is really applied in my environment?
>
> Even tried 0003, but no change.
>
> Patch expectedly doesn't apply on 9.3.6:
>
> [root@zivwebapp13 postgresql-9.3.6patched]# patch -p1
> <../0001-Fix-sslv3-alert-unexpected-message-errors-in-SSL-ren.patch patching
> file src/interfaces/libpq/fe-misc.c
> Hunk #1 succeeded at 919 (offset -1 lines).
> can't find file to patch at input line 45
> Perhaps you used the wrong -p or --strip option?
> The text leading up to this was:
> --------------------------
> |diff --git a/src/interfaces/libpq/fe-secure-openssl.c
> b/src/interfaces/libpq/fe-secure-openssl.c
> |index a32af34..93b8184 100644
> |--- a/src/interfaces/libpq/fe-secure-openssl.c
> |+++ b/src/interfaces/libpq/fe-secure-openssl.c
> --------------------------
>
> I would also test backpatched patch code once it's made.
>
> VlG
>
> Arne
>
>
> On Sat, 14 Feb 2015, Heikki Linnakangas wrote:
>
>> On 02/13/2015 10:59 PM, Andres Freund wrote:
>>> On 2015-02-13 18:52:02 +0000, pilum.70@uni-muenster.de wrote:
>>>> I get ssl renegotiation failures with streaming standbys. Sometimes the
>>>> connection breaks and is reconnected afterwards. However, if I use
>>>> pg_basebackup (same libpq connection string), I don't get any of these
>>>> failures, allthough the transferred data ist far beyond 512 MB
>>>> So I don't think it's the
>>>> ssl renegotiation bug (openssl of a yum update patched centos6)
>>>> If I disable ssl_renegotiation_limit to 0, there are no errors any more,
>>>> but that is only a workaround, no solution.
>>>
>>> Heikki and me have recently investigated problems around SSL
>>> renegotiation. See
>>> http://www.postgresql.org/message-id/20150126101405.GA31719@awork2.anarazel.de
>>> .
>>
>> I wasn't able to reproduce exactly the same error you saw, Arne, so it
>> would be good if you could test the patches I've been developing, to see if
>> they fix your problem too. That is, patches 0001 and 0002 from
>> http://www.postgresql.org/message-id/54DE6FAF.6050005@vmware.com. Could you
>> do that?
>>
>> - Heikki
>>
>

Re: BUG #12769: SSL-Renegotiation failures

From
Heikki Linnakangas
Date:
On 02/19/2015 05:06 PM, Arne Scheffer wrote:
> Retried my test suite after commit
>
> 1c2b7c0879d83ff79e4adf2c0a883df92b713da4     Restore the SSL_set_session_id_context() call to OpenSS...
>
> Cloned again from 9.5devel master.
> Now the two patches work as expected (Means: errors without patch, no errors
> with patch (I can't see, whether the code is really performing renegotiation)).
> That seems promising.
>
> As already mentioned:
> I would also test the two patches for 9.3
> in a production near environment to prove them fixing the reported Bug.
> If that isn't desired,
> I will clone them from REL9_3_STABLE branch once they are committed.

Thanks, and sorry for not responding sooner. I investigated this on
Tuesday, and committed that fix. I was able to reproduce the problem you
saw on a Centos 6 VM, and also concluded that it now works on master
with those two patches applied. I tried backpatching them to 9.4, but I
still saw some errors on 9.4 even with the patches. I ran out of time to
dig deepeer, but it seems that some of the other changes done in master
are needed to fully fix this. There has been a lot of changes between
9.4 and master, and some changes between 9.3 and 9.4 as well, so it's
probably going to take some time to come up with correct fixes for all
the back-branches.

- Heikki

Re: BUG #12769: SSL-Renegotiation failures

From
Arne Scheffer
Date:
Hi,

On Thu, 19 Feb 2015, Heikki Linnakangas wrote:

> On 02/19/2015 05:06 PM, Arne Scheffer wrote:
>> Retried my test suite after commit
>>
>> 1c2b7c0879d83ff79e4adf2c0a883df92b713da4     Restore the
>> SSL_set_session_id_context() call to OpenSS...
>>
>> Cloned again from 9.5devel master.
>> Now the two patches work as expected (Means: errors without patch, no
>> errors
>> with patch (I can't see, whether the code is really performing
>> renegotiation)).
>> That seems promising.
>>
>> As already mentioned:
>> I would also test the two patches for 9.3
>> in a production near environment to prove them fixing the reported Bug.
>> If that isn't desired,
>> I will clone them from REL9_3_STABLE branch once they are committed.
>
> Thanks, and sorry for not responding sooner. I investigated this on Tuesday,
> and committed that fix. I was able to reproduce the problem you saw on a
> Centos 6 VM, and also concluded that it now works on master with those two
> patches applied. I tried backpatching them to 9.4, but I still saw some
> errors on 9.4 even with the patches. I ran out of time to dig deepeer, but it
> seems that some of the other changes done in master are needed to fully fix
> this. There has been a lot of changes between 9.4 and master, and some
> changes between 9.3 and 9.4 as well, so it's probably going to take some time
> to come up with correct fixes for all the back-branches.

Any progress fixing this bug?
If not, wouldn't it make sense to (at least) include the Fixes into 9.5?

VlG

Arne Scheffer