Thread: disable SSL compression?

It appears that SSL compression is nowadays deprecated as insecure.
Yet, it is still enabled by libpq by default, and there is no way to
disable it in the server.  Should we make some changes here?  Does
anyone know more about this?

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Thu, Mar 8, 2018 at 3:40 PM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:
> It appears that SSL compression is nowadays deprecated as insecure.
> Yet, it is still enabled by libpq by default, and there is no way to
> disable it in the server.  Should we make some changes here?  Does
> anyone know more about this?

Even if libpq enables it, it has to be enabled both in the client and
the server for it to work.

OpenSSL disables the whole feature by default, and enabling it is
rather cumbersome. The result is that, at least with OpenSSL, the
server and client won't accept compression without extensive fiddling
by the user.

So I don't think libpq has to change anything here.

On 3/8/18 14:23, Claudio Freire wrote:
> On Thu, Mar 8, 2018 at 3:40 PM, Peter Eisentraut
> <peter.eisentraut@2ndquadrant.com> wrote:
>> It appears that SSL compression is nowadays deprecated as insecure.
>> Yet, it is still enabled by libpq by default, and there is no way to
>> disable it in the server.  Should we make some changes here?  Does
>> anyone know more about this?
> 
> Even if libpq enables it, it has to be enabled both in the client and
> the server for it to work.
> 
> OpenSSL disables the whole feature by default, and enabling it is
> rather cumbersome. The result is that, at least with OpenSSL, the
> server and client won't accept compression without extensive fiddling
> by the user.

But however that may be, libpq appears to enable it by default.  This is
what I get from psql:

SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384,
bits: 256, compression: on)

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Thu, Mar 8, 2018 at 11:06 PM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:
> On 3/8/18 14:23, Claudio Freire wrote:
>> On Thu, Mar 8, 2018 at 3:40 PM, Peter Eisentraut
>> <peter.eisentraut@2ndquadrant.com> wrote:
>>> It appears that SSL compression is nowadays deprecated as insecure.
>>> Yet, it is still enabled by libpq by default, and there is no way to
>>> disable it in the server.  Should we make some changes here?  Does
>>> anyone know more about this?
>>
>> Even if libpq enables it, it has to be enabled both in the client and
>> the server for it to work.
>>
>> OpenSSL disables the whole feature by default, and enabling it is
>> rather cumbersome. The result is that, at least with OpenSSL, the
>> server and client won't accept compression without extensive fiddling
>> by the user.
>
> But however that may be, libpq appears to enable it by default.  This is
> what I get from psql:
>
> SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384,
> bits: 256, compression: on)

I don't get that:

SSL connection (protocol: TLSv1.2, cipher:
ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)

Even if I set OPENSSL_DEFAULT_ZLIB=1 on the client, I get the same.
The serverside refuses.

I'm totally unconvinced by the threat posed by exploiting a client by tricking it into requesting protocol compression - or any other protocol change the client lib doesn't understand - with a connection option in PGOPTIONS or the "options" connstring entry. The attacker must be able to specify either environment variables (in which case I present "LD_PRELOAD") or the connstr. If they can set a connstr they can direct the client to talk to a different host that tries to exploit the connecting client in whatever manner they wish by sending any custom crafted messages they like.

If the attacker has access to client process or environment, he's already won and this is not where the compression vulnerability lies.

CRIME and BREACH attacks with (SSL) compression are known plaintext attacks, which require the attacker 1) to have ability to observe encrypted data and 2) have a way to influence the plain text, in this case SQL query. In the case of CRIME HTTPS attack, compression state was shared between page content and request headers, thus by observing size of responses, which are in HTTP headers, one could guess cookie values and steal credentials even though the javascript making requests was running on different domain.

So the vulnerability would be in guessing some values in request or response, which the application or protocol might want to keep hidden, while somehow getting the size of request or response from database. Thus, sharing compression state too widely might not be wise.

Kind regards,
Gasper

On 3/9/18 09:06, Magnus Hagander wrote:
> What platform does that actually work out of the box on? I have
> customers who actively want to use it (for compression, not security --
> replication across limited and metered links), and the amount of
> workarounds they have to put in place OS level to get it working is
> increasingly complicated.

It was disabled in OpenSSL 1.1.0:

  *) CRIME protection: disable compression by default, even if OpenSSL is
     compiled with zlib enabled. Applications can still enable compression
     by calling SSL_CTX_clear_options(ctx, SSL_OP_NO_COMPRESSION), or by
     using the SSL_CONF library to configure compression.
     [Emilia Käsper]

So for your purposes, you could add a server option to turn it back on.

Such a server option would also be useful for those users who are using
OpenSSL <1.1.0 and want to turn off compression on the server side.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 3/11/18 04:00, Magnus Hagander wrote:
> I am not talking about the OpenSSL disabling it. It was disabled on most
> *distributions* years ago, long before that commit. Which is why I'm
> still curious as to what platform you actually got it enabled by default
> on...

Homebrew package

>     So for your purposes, you could add a server option to turn it back on.
> 
>     Such a server option would also be useful for those users who are using
>     OpenSSL <1.1.0 and want to turn off compression on the server side.
> 
> 
> We'd probably have to put in the distribution specific workarounds like
> mentioned above to make it actually useful for that. 

The change in the Debian package I found was to build without zlib at
all.  So no amount of turning it back on will help.  Whereas the
upstream change was just to make the default to be off.  But anyway,
this feature is clearly dying, so we probably shouldn't be trying very
hard to keep it.

My proposal is the attached patch that sets the default in libpq to off
and adjusts the documentation a bit so it doesn't sound like we have
missed the news altogether.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> The change in the Debian package I found was to build without zlib at
> all.  So no amount of turning it back on will help.  Whereas the
> upstream change was just to make the default to be off.  But anyway,
> this feature is clearly dying, so we probably shouldn't be trying very
> hard to keep it.

Right; the other point made in the referenced thread was that future
versions of the TLS spec would probably drop compression altogether.
So adding any sort of server or libpq option is work that's going to
be wasted in the long run.

> My proposal is the attached patch that sets the default in libpq to off
> and adjusts the documentation a bit so it doesn't sound like we have
> missed the news altogether.

Seems reasonable as far as it goes, but do we need to make corresponding
server-side changes?

            regards, tom lane

On 3/11/18 12:07, Magnus Hagander wrote:
> I think it's worth mentioning in the docs around "it's now considered
> insecure" that it's still an option to use if compression is the main
> thing one is looking for, rather than security. As in, it doesn't make
> it any less secure than no ssl at all. (obviously not those words)
> 
> +1 otherwise. 

committed like that

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 3/11/18 13:28, Tom Lane wrote:
>> My proposal is the attached patch that sets the default in libpq to off
>> and adjusts the documentation a bit so it doesn't sound like we have
>> missed the news altogether.
> 
> Seems reasonable as far as it goes, but do we need to make corresponding
> server-side changes?

We could add a setting to disable SSL compression on the server, as some
web servers have been doing, but the niche of version combinations where
that would be applicable seems pretty low.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 3/11/18 13:28, Tom Lane wrote:

My proposal is the attached patch that sets the default in libpq to off
and adjusts the documentation a bit so it doesn't sound like we have
missed the news altogether.

Seems reasonable as far as it goes, but do we need to make corresponding
server-side changes?

We could add a setting to disable SSL compression on the server, as some
web servers have been doing, but the niche of version combinations where
that would be applicable seems pretty low.

One of our customers was managed to improve speed about 10 times by using SSL compression for the system where client and servers are located in different geographical regions
and query results are very large because of JSON columns. Them actually do not need encryption, just compression.
I expect that it is not the only case where compression of libpq protocol can be useful. Please notice that logical replication is also using libpq protocol.

If SSL compression is deprecated, should we provide own compression?
I have implemented some prototype implementation of it (patch is attached).
I have added compression=on/off parameter to connection string and -Z option to psql and pgbench utilities.
Below are some results:

Compression ratio (raw->compressed):

	libz (level=1)	libzstd (level=1)
pgbench -i -s 10	16997209->2536330	16997209->268077
pgbench -t 100000 -S	6289036->1523862 6600338<-900293	6288933->1777400 6600338<-1000318

There is no mistyping: libzstd compress COPY data about 10 times better than libz, with wonderful compression ratio 63.

Influence on execution time is minimal (I have tested local configuration when client and server are at the same host):

	no compression	libz (level=1)	libzstd (level=1)
pgbench -i -s 10	1.552	1.572	1.611
pgbench -t 100000 -S	4.482	4.926	4.877

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

On 17.03.2018 17:18, Peter Eisentraut wrote:

On 3/11/18 13:28, Tom Lane wrote:

My proposal is the attached patch that sets the default in libpq to off
and adjusts the documentation a bit so it doesn't sound like we have
missed the news altogether.

Seems reasonable as far as it goes, but do we need to make corresponding
server-side changes?

We could add a setting to disable SSL compression on the server, as some
web servers have been doing, but the niche of version combinations where
that would be applicable seems pretty low.

One of our customers was managed to improve speed about 10 times by using SSL compression for the system where client and servers are located in different geographical regions
and query results are very large because of JSON columns. Them actually do not need encryption, just compression.
I expect that it is not the only case where compression of libpq protocol can be useful. Please notice that logical replication is also using libpq protocol.

If SSL compression is deprecated, should we provide own compression?
I have implemented some prototype implementation of it (patch is attached).
I have added compression=on/off parameter to connection string and -Z option to psql and pgbench utilities.
Below are some results:

Compression ratio (raw->compressed):

	libz (level=1)	libzstd (level=1)
pgbench -i -s 10	16997209->2536330	16997209->268077
pgbench -t 100000 -S	6289036->1523862 6600338<-900293	6288933->1777400 6600338<-1000318

There is no mistyping: libzstd compress COPY data about 10 times better than libz, with wonderful compression ratio 63.

Influence on execution time is minimal (I have tested local configuration when client and server are at the same host):

	no compression	libz (level=1)	libzstd (level=1)
pgbench -i -s 10	1.552	1.572	1.611
pgbench -t 100000 -S	4.482	4.926	4.877

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

Sorry, I have attached wrong patch.
To use zstd compression, Postgres should be configured with --with-zstd. Otherwise compression will use zlib unless it is disabled by --without-zlib option.

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

On 3/28/18 13:26, Konstantin Knizhnik wrote:
> If SSL compression is deprecated, should we provide own compression?
> I have implemented some prototype implementation of it (patch is attached).
> I have added compression=on/off parameter to connection string and -Z
> option to psql and pgbench utilities.

What I'd like to see here is extensive protocol documentation that
describes the compression method negotiation, and the interaction with
SSL, and a test suite to support that.

Maybe start a new thread.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On March 28, 2018 4:15:02 PM PDT, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote:
>On 3/28/18 13:26, Konstantin Knizhnik wrote:
>> If SSL compression is deprecated, should we provide own compression?
>> I have implemented some prototype implementation of it (patch is
>attached).
>> I have added compression=on/off parameter to connection string and -Z
>> option to psql and pgbench utilities.
>
>What I'd like to see here is extensive protocol documentation that
>describes the compression method negotiation, and the interaction with
>SSL, and a test suite to support that.
>
>Maybe start a new thread.

+analysis of whether that's safe to do from a cryptographic POV. There's a reason compression has been disabled for
justabout all SSL/TLS libraries. 

Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

On Wed, Mar 28, 2018 at 7:16 PM, Andres Freund <andres@anarazel.de> wrote:
> +analysis of whether that's safe to do from a cryptographic POV. There's a reason compression has been disabled for
justabout all SSL/TLS libraries.
 

As I understand it on a brief review of the Google search
results^W^W^Wliterature, the reason that was done was to prevent
things like the CRIME attack, which apparently involves Javascript
running in your browser from deducing information that it shouldn't be
able to get, like the Cookies that are sent along with the requests it
initiates.  No similar attack should be possible against PostgreSQL
because there's no similar kind of privilege separation.  Your PG
driver doesn't have untrusted Javascript running inside of it, we
hope.

In general, I'd expect compressing data to be beneficial for the
security of encryption because it should increase the entropy of the
encrypted bytes, but obviously it's not hard to hypothesize cases
where the opposite is true for one reason or another.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 4/2/18 10:25, Robert Haas wrote:
> As I understand it on a brief review of the Google search
> results^W^W^Wliterature, the reason that was done was to prevent
> things like the CRIME attack, which apparently involves Javascript
> running in your browser from deducing information that it shouldn't be
> able to get, like the Cookies that are sent along with the requests it
> initiates.  No similar attack should be possible against PostgreSQL
> because there's no similar kind of privilege separation.  Your PG
> driver doesn't have untrusted Javascript running inside of it, we
> hope.

I think the attack is much more general involving two server end points,
one of which is under the control of the attacker and provides
information that the client is using to query the second server,
independent of the protocols.

For example, if your application code does maybe a geoip lookup and then
does

select * from sometable
    where a = $geo_data and b = 'secret bank account number';

then the operator of the geoip service (or someone impersonating it, of
course) can just rotate the lookup results through the bank account
number space until they notice that the compression result changes, in
which case they have matched the bank account number.

In the web space, that is easier because the client code is typically
viewable by the attacker, and this kind of query is more common (the
"bank account number" is typically a session key), but the principle is
the same.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Mon, Apr 2, 2018 at 10:52 AM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:
> On 4/2/18 10:25, Robert Haas wrote:
>> As I understand it on a brief review of the Google search
>> results^W^W^Wliterature, the reason that was done was to prevent
>> things like the CRIME attack, which apparently involves Javascript
>> running in your browser from deducing information that it shouldn't be
>> able to get, like the Cookies that are sent along with the requests it
>> initiates.  No similar attack should be possible against PostgreSQL
>> because there's no similar kind of privilege separation.  Your PG
>> driver doesn't have untrusted Javascript running inside of it, we
>> hope.
>
> I think the attack is much more general involving two server end points,
> one of which is under the control of the attacker and provides
> information that the client is using to query the second server,
> independent of the protocols.
>
> For example, if your application code does maybe a geoip lookup and then
> does
>
> select * from sometable
>     where a = $geo_data and b = 'secret bank account number';
>
> then the operator of the geoip service (or someone impersonating it, of
> course) can just rotate the lookup results through the bank account
> number space until they notice that the compression result changes, in
> which case they have matched the bank account number.
>
> In the web space, that is easier because the client code is typically
> viewable by the attacker, and this kind of query is more common (the
> "bank account number" is typically a session key), but the principle is
> the same.

Ah.  Yeah, that makes sense.  Although to use PG as a vector, it seems
like the attacker would need to the ability to snoop network traffic
between the application server and PG.  If both of those are
presumably inside the bank's network and yet an attacker can sniff
them, to some degree you've already lost.  Now it could be that a
rogue bank employee is trying to gain access from within the bank, or
maybe your bank exposes application-to-database traffic on the public
Internet.  But in general that seems like traffic that should be
well-secured anyway for lots of reasons, as opposed to the case where
one part of your browser is trying to hide information from another
part of your browser, which is a lot harder to isolate thoroughly.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 4/2/18 11:05, Robert Haas wrote:
> Ah.  Yeah, that makes sense.  Although to use PG as a vector, it seems
> like the attacker would need to the ability to snoop network traffic
> between the application server and PG.  If both of those are
> presumably inside the bank's network and yet an attacker can sniff
> them, to some degree you've already lost.  Now it could be that a
> rogue bank employee is trying to gain access from within the bank, or
> maybe your bank exposes application-to-database traffic on the public
> Internet.  But in general that seems like traffic that should be
> well-secured anyway for lots of reasons, as opposed to the case where
> one part of your browser is trying to hide information from another
> part of your browser, which is a lot harder to isolate thoroughly.

I agree the attack is less likely to be applicable in typical database
installations.  I think we should move forward with considering protocol
compression proposals, but any final result should put a warning in the
documentation that using compression is potentially insecure.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> I agree the attack is less likely to be applicable in typical database
> installations.  I think we should move forward with considering protocol
> compression proposals, but any final result should put a warning in the
> documentation that using compression is potentially insecure.

It seemed like the attack you described wasn't all that dependent on
whether the data is compressed or not: if you can see the size of the
server's reply to "select ... where account_number = x", you can pretty
well tell the difference between 0 and 1 rows, with or without
compression.  So I'm still not very clear on what the threat model is.

            regards, tom lane

On 4/2/18 12:46, Tom Lane wrote:
> Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
>> I agree the attack is less likely to be applicable in typical database
>> installations.  I think we should move forward with considering protocol
>> compression proposals, but any final result should put a warning in the
>> documentation that using compression is potentially insecure.
> 
> It seemed like the attack you described wasn't all that dependent on
> whether the data is compressed or not: if you can see the size of the
> server's reply to "select ... where account_number = x", you can pretty
> well tell the difference between 0 and 1 rows, with or without
> compression.  So I'm still not very clear on what the threat model is.

Well these could also be update commands or procedure calls with a
constant response size.  Also, it doesn't matter whether the select
returns anything.  Maybe it's not querying the main accounts table.  But
it already shows that the client thinks that the account number is a
real one.

There are probably even better examples.  But the main point is that if
an attacker can control part of what you send alongside some secret
thing, compression is going to be a security concern for some.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Mon, Apr 02, 2018 at 12:46:25PM -0400, Tom Lane wrote:
> Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> > I agree the attack is less likely to be applicable in typical database
> > installations.  I think we should move forward with considering protocol
> > compression proposals, but any final result should put a warning in the
> > documentation that using compression is potentially insecure.
> 
> It seemed like the attack you described wasn't all that dependent on
> whether the data is compressed or not: 

I think it is.  
I wrote something longer about this in 2013.
https://www.postgresql.org/message-id/20130115172253.GA6662@isc.upenn.edu

The key idea is you can now to a radix search of a secret rather than a
single 1-bit match/doesn't match oracle type attack which is not
feasible when you have an high entropy low density value like a big
ascii number or hex string.

I do something like set my preferred email (or anything an application
lets a user change) to something like a3bf and keep extending the guess
as I go and sometimes I compress along with the secret and rather than
have to guess a whole application session id all at once and have a work
factor of O(2^128) or something I've got a very tractable search.

It might not be something that a problem for everyone, but in some
situations it's attackable.

Garick

Hi,

On 2018-04-02 10:25:04 -0400, Robert Haas wrote:
> In general, I'd expect compressing data to be beneficial for the
> security of encryption because it should increase the entropy of the
> encrypted bytes, but obviously it's not hard to hypothesize cases
> where the opposite is true for one reason or another.

I don't think it's actually ever a really positive thing for security to
compress before encrypting, and encrypting after should always be
useless.  The problem is that that opens one hell of a sidechannel
attack, because you're suddenly leaking information about the
compressability of the transferred data. If there's any way attackers
have knowledge, or worse influence, of any of the transported data that
allows to make inferrerences about the content and potentially the key.

Whereas there should never be a observable difference in the encrypted
stream, if you use a sane cipher mode (i.e. NOT ECB).

Greetings,

Andres Freund

Hi,

On 2018-04-02 12:46:25 -0400, Tom Lane wrote:
> It seemed like the attack you described wasn't all that dependent on
> whether the data is compressed or not: if you can see the size of the
> server's reply to "select ... where account_number = x", you can pretty
> well tell the difference between 0 and 1 rows, with or without
> compression.  So I'm still not very clear on what the threat model is.

Imagine that the attacker has control over *parts* of a query. The
server always sends a query like
SELECT mysecret FROM secrets WHERE ...;
but somehow, and there's definitely sneaky ways, there's a second query
sent alongside.
SELECT '$user-defined-string';

If those are compressed together *and* the attacker can observe the size
of the returned result, the attacker can change $user-defined-string
iteratively and infer knowledge about what the contents of mysecret
are. If $user-defined-string and mysecret are the same you're going to
get a smaller total response packet.

One such injection vector that's commonly changable can just be a
username or such.

Greetings,

Andres Freund