Thread: BUG #12769: SSL-Renegotiation failures
The following bug has been logged on the website: Bug reference: 12769 Logged by: Arne Scheffer Email address: pilum.70@uni-muenster.de PostgreSQL version: 9.3.6 Operating system: CentOS6, openssl version OpenSSL 1.0.1e-fips Description: I get ssl renegotiation failures with streaming standbys. Sometimes the connection breaks and is reconnected afterwards. However, if I use pg_basebackup (same libpq connection string), I don't get any of these failures, allthough the transferred data ist far beyond 512 MB So I don't think it's the ssl renegotiation bug (openssl of a yum update patched centos6) If I disable ssl_renegotiation_limit to 0, there are no errors any more, but that is only a workaround, no solution. Steps to reproduce with testdata: Tested on patched centos6, Postgresql 9.3.6 bash-4.1$ openssl version OpenSSL 1.0.1e-fips 11 Feb 2013 Install Repository yum install postgresql93 postgresql93-server postgresql93-contrib postgresql93-devel postgresql93-libs /usr/pgsql-9.3/bin/initdb -D /tmp/data1 Not necessary but comfortable: create two postgresql-93test Init-Scripts (cp /etc/init.d/postgresql-9.3 postgresql-93test) - changing data-Directory to /tmp/data1 und /tmp/data2 - changing PGPORT to yourport and yourport2 Start server in data1 On server in data1: create user replicateme with replication encrypted password "insecure"; <##### postgresql.conf: ssl = on listen_addresses = 'yourserver.domain.com' wal_level = hot_standby # 4GB fuer WAL-Segmente: wal_keep_segments = 256 max_wal_senders = 2 #####> Certificates installed <##### only 1 line: pg_hba.conf: hostssl replication ... #####> usr/pgsql-9.3/bin/pg_basebackup -D /tmp/data2 -v -P -Fp -l testsslbackup -x -d "host=yourserver.domain.com port=yourport user=replicator sslmode=verify-full" <#### recovery.conf: recovery_target_timeline = 'latest' standby_mode = 'on' primary_conninfo = 'host=yourserver.domain.com port=yourport user=replicator sslmode=verify-full' #####> Start standby server in data2 On server in data1: create database test \c test create table test2 (a int, b text); insert into test2 (select generate_series(0,5000000,1)); update test2 set b='sljfdlksjaofjsaolfjsdlkofjdolasjfoldsjfoldsjaolfjdloksjfolajsdflkjsdlokfjslojdalkfjoldsjfoslafjolasljfdloas'; tail -f /tmp/data1/pg_log/postgresql-Fri.log [...] < 2015-02-13 18:09:55.964 CET >LOG: SSL renegotiation failure [...] < 2015-02-13 18:10:11.819 CET >LOG: SSL renegotiation failure [...] usr/pgsql-9.3/bin/pg_basebackup -D /tmp/data3 -v -P -Fp -l testsslbackup2 -x -d "host=yourserver.domain.com port=yourport user=replicator sslmode=verify-full" tail -f /tmp/data1/pg_log/postgresql-Fri.log --> nothing!
Hi Arne, On 2015-02-13 18:52:02 +0000, pilum.70@uni-muenster.de wrote: > The following bug has been logged on the website: > > Bug reference: 12769 > Logged by: Arne Scheffer > Email address: pilum.70@uni-muenster.de > PostgreSQL version: 9.3.6 > Operating system: CentOS6, openssl version OpenSSL 1.0.1e-fips > Description: > > I get ssl renegotiation failures with streaming standbys. Sometimes the > connection breaks and is reconnected afterwards. However, if I use > pg_basebackup (same libpq connection string), I don't get any of these > failures, allthough the transferred data ist far beyond 512 MB > So I don't think it's the > ssl renegotiation bug (openssl of a yum update patched centos6) > If I disable ssl_renegotiation_limit to 0, there are no errors any more, > but that is only a workaround, no solution. Heikki and me have recently investigated problems around SSL renegotiation. See http://www.postgresql.org/message-id/20150126101405.GA31719@awork2.anarazel.de . Greetings, Andres Freund
On 02/13/2015 10:59 PM, Andres Freund wrote: > On 2015-02-13 18:52:02 +0000, pilum.70@uni-muenster.de wrote: >> I get ssl renegotiation failures with streaming standbys. Sometimes the >> connection breaks and is reconnected afterwards. However, if I use >> pg_basebackup (same libpq connection string), I don't get any of these >> failures, allthough the transferred data ist far beyond 512 MB >> So I don't think it's the >> ssl renegotiation bug (openssl of a yum update patched centos6) >> If I disable ssl_renegotiation_limit to 0, there are no errors any more, >> but that is only a workaround, no solution. > > Heikki and me have recently investigated problems around SSL > renegotiation. See > http://www.postgresql.org/message-id/20150126101405.GA31719@awork2.anarazel.de > . I wasn't able to reproduce exactly the same error you saw, Arne, so it would be good if you could test the patches I've been developing, to see if they fix your problem too. That is, patches 0001 and 0002 from http://www.postgresql.org/message-id/54DE6FAF.6050005@vmware.com. Could you do that? - Heikki
On the same machine (CentOS6) Cloned from 9.5devel, ./configure --with-perl --with-openssl --with-python --with-tcl --with-pam --with-ldap --enable-thread-safety --enable-debug make make install make clean Repeated the procedure attached in the mail Got a different error (also twice at exected renegotiation times): < 2015-02-15 16:40:45.438 CET >LOG: SSL error: session id context uninitialized < 2015-02-15 16:40:45.439 CET >LOG: could not receive data from client: Connection reset by peer < 2015-02-15 16:40:45.439 CET >LOG: unexpected EOF on standby connection Tried git checkout -b ssl_patch patch -p1 <../0001-Fix-sslv3-alert-unexpected-message-errors-in-SSL-ren.patch (got applied on 2 files) patch -p1 <../0002-Also-drain-input-buffer-in-non-blocking-mode-if-send.patch (got applied on 1 file) Repeated make make install make clean Repeated the procedure attached in the mail. (Both twice.) Got the same errors. Perhaps I did something wrong. Could you add a temporary debug line, so that I can see, that the patch is really applied in my environment? Even tried 0003, but no change. Patch expectedly doesn't apply on 9.3.6: [root@zivwebapp13 postgresql-9.3.6patched]# patch -p1 <../0001-Fix-sslv3-alert-unexpected-message-errors-in-SSL-ren.patch patching file src/interfaces/libpq/fe-misc.c Hunk #1 succeeded at 919 (offset -1 lines). can't find file to patch at input line 45 Perhaps you used the wrong -p or --strip option? The text leading up to this was: -------------------------- |diff --git a/src/interfaces/libpq/fe-secure-openssl.c b/src/interfaces/libpq/fe-secure-openssl.c |index a32af34..93b8184 100644 |--- a/src/interfaces/libpq/fe-secure-openssl.c |+++ b/src/interfaces/libpq/fe-secure-openssl.c -------------------------- I would also test backpatched patch code once it's made. VlG Arne On Sat, 14 Feb 2015, Heikki Linnakangas wrote: > On 02/13/2015 10:59 PM, Andres Freund wrote: >> On 2015-02-13 18:52:02 +0000, pilum.70@uni-muenster.de wrote: >>> I get ssl renegotiation failures with streaming standbys. Sometimes the >>> connection breaks and is reconnected afterwards. However, if I use >>> pg_basebackup (same libpq connection string), I don't get any of these >>> failures, allthough the transferred data ist far beyond 512 MB >>> So I don't think it's the >>> ssl renegotiation bug (openssl of a yum update patched centos6) >>> If I disable ssl_renegotiation_limit to 0, there are no errors any more, >>> but that is only a workaround, no solution. >> >> Heikki and me have recently investigated problems around SSL >> renegotiation. See >> http://www.postgresql.org/message-id/20150126101405.GA31719@awork2.anarazel.de >> . > > I wasn't able to reproduce exactly the same error you saw, Arne, so it would > be good if you could test the patches I've been developing, to see if they > fix your problem too. That is, patches 0001 and 0002 from > http://www.postgresql.org/message-id/54DE6FAF.6050005@vmware.com. Could you > do that? > > - Heikki >
Retried my test suite after commit 1c2b7c0879d83ff79e4adf2c0a883df92b713da4 Restore the SSL_set_session_id_context() call to OpenSS... Cloned again from 9.5devel master. Now the two patches work as expected (Means: errors without patch, no errors with patch (I can't see, whether the code is really performing renegotiation)). That seems promising. As already mentioned: I would also test the two patches for 9.3 in a production near environment to prove them fixing the reported Bug. If that isn't desired, I will clone them from REL9_3_STABLE branch once they are committed. VlG-Arne On Sun, 15 Feb 2015, Arne Scheffer wrote: > On the same machine (CentOS6) > > Cloned from 9.5devel, > ./configure --with-perl --with-openssl --with-python --with-tcl --with-pam > --with-ldap --enable-thread-safety --enable-debug > make make install make clean > > Repeated the procedure attached in the mail > Got a different error (also twice at exected renegotiation times): > > < 2015-02-15 16:40:45.438 CET >LOG: SSL error: session id context > uninitialized > < 2015-02-15 16:40:45.439 CET >LOG: could not receive data from client: > Connection reset by peer > < 2015-02-15 16:40:45.439 CET >LOG: unexpected EOF on standby connection > > Tried > > git checkout -b ssl_patch > patch -p1 <../0001-Fix-sslv3-alert-unexpected-message-errors-in-SSL-ren.patch > (got applied on 2 files) > patch -p1 <../0002-Also-drain-input-buffer-in-non-blocking-mode-if-send.patch > (got applied on 1 file) > > Repeated make make install make clean > > Repeated the procedure attached in the mail. > > (Both twice.) > > Got the same errors. > Perhaps I did something wrong. Could you add a temporary debug line, so that > I can > see, that the patch is really applied in my environment? > > Even tried 0003, but no change. > > Patch expectedly doesn't apply on 9.3.6: > > [root@zivwebapp13 postgresql-9.3.6patched]# patch -p1 > <../0001-Fix-sslv3-alert-unexpected-message-errors-in-SSL-ren.patch patching > file src/interfaces/libpq/fe-misc.c > Hunk #1 succeeded at 919 (offset -1 lines). > can't find file to patch at input line 45 > Perhaps you used the wrong -p or --strip option? > The text leading up to this was: > -------------------------- > |diff --git a/src/interfaces/libpq/fe-secure-openssl.c > b/src/interfaces/libpq/fe-secure-openssl.c > |index a32af34..93b8184 100644 > |--- a/src/interfaces/libpq/fe-secure-openssl.c > |+++ b/src/interfaces/libpq/fe-secure-openssl.c > -------------------------- > > I would also test backpatched patch code once it's made. > > VlG > > Arne > > > On Sat, 14 Feb 2015, Heikki Linnakangas wrote: > >> On 02/13/2015 10:59 PM, Andres Freund wrote: >>> On 2015-02-13 18:52:02 +0000, pilum.70@uni-muenster.de wrote: >>>> I get ssl renegotiation failures with streaming standbys. Sometimes the >>>> connection breaks and is reconnected afterwards. However, if I use >>>> pg_basebackup (same libpq connection string), I don't get any of these >>>> failures, allthough the transferred data ist far beyond 512 MB >>>> So I don't think it's the >>>> ssl renegotiation bug (openssl of a yum update patched centos6) >>>> If I disable ssl_renegotiation_limit to 0, there are no errors any more, >>>> but that is only a workaround, no solution. >>> >>> Heikki and me have recently investigated problems around SSL >>> renegotiation. See >>> http://www.postgresql.org/message-id/20150126101405.GA31719@awork2.anarazel.de >>> . >> >> I wasn't able to reproduce exactly the same error you saw, Arne, so it >> would be good if you could test the patches I've been developing, to see if >> they fix your problem too. That is, patches 0001 and 0002 from >> http://www.postgresql.org/message-id/54DE6FAF.6050005@vmware.com. Could you >> do that? >> >> - Heikki >> >
On 02/19/2015 05:06 PM, Arne Scheffer wrote: > Retried my test suite after commit > > 1c2b7c0879d83ff79e4adf2c0a883df92b713da4 Restore the SSL_set_session_id_context() call to OpenSS... > > Cloned again from 9.5devel master. > Now the two patches work as expected (Means: errors without patch, no errors > with patch (I can't see, whether the code is really performing renegotiation)). > That seems promising. > > As already mentioned: > I would also test the two patches for 9.3 > in a production near environment to prove them fixing the reported Bug. > If that isn't desired, > I will clone them from REL9_3_STABLE branch once they are committed. Thanks, and sorry for not responding sooner. I investigated this on Tuesday, and committed that fix. I was able to reproduce the problem you saw on a Centos 6 VM, and also concluded that it now works on master with those two patches applied. I tried backpatching them to 9.4, but I still saw some errors on 9.4 even with the patches. I ran out of time to dig deepeer, but it seems that some of the other changes done in master are needed to fully fix this. There has been a lot of changes between 9.4 and master, and some changes between 9.3 and 9.4 as well, so it's probably going to take some time to come up with correct fixes for all the back-branches. - Heikki
Hi, On Thu, 19 Feb 2015, Heikki Linnakangas wrote: > On 02/19/2015 05:06 PM, Arne Scheffer wrote: >> Retried my test suite after commit >> >> 1c2b7c0879d83ff79e4adf2c0a883df92b713da4 Restore the >> SSL_set_session_id_context() call to OpenSS... >> >> Cloned again from 9.5devel master. >> Now the two patches work as expected (Means: errors without patch, no >> errors >> with patch (I can't see, whether the code is really performing >> renegotiation)). >> That seems promising. >> >> As already mentioned: >> I would also test the two patches for 9.3 >> in a production near environment to prove them fixing the reported Bug. >> If that isn't desired, >> I will clone them from REL9_3_STABLE branch once they are committed. > > Thanks, and sorry for not responding sooner. I investigated this on Tuesday, > and committed that fix. I was able to reproduce the problem you saw on a > Centos 6 VM, and also concluded that it now works on master with those two > patches applied. I tried backpatching them to 9.4, but I still saw some > errors on 9.4 even with the patches. I ran out of time to dig deepeer, but it > seems that some of the other changes done in master are needed to fully fix > this. There has been a lot of changes between 9.4 and master, and some > changes between 9.3 and 9.4 as well, so it's probably going to take some time > to come up with correct fixes for all the back-branches. Any progress fixing this bug? If not, wouldn't it make sense to (at least) include the Fixes into 9.5? VlG Arne Scheffer