Thread: Disable OpenSSL compression
I ran into a performance problem described in this thread: http://archives.postgresql.org/pgsql-performance/2011-10/msg00249.php continued here: http://archives.postgresql.org/pgsql-performance/2011-11/msg00045.php OpenSSL compresses data by default, and that causes a performance penalty of 100% and more, at least when SELECTing larger bytea objects. The backend process becomes CPU bound. >From OpenSSL version 1.0.0. on, compression can be disabled. The attached patch does that, and with that patch I see dramatic performance improvements: Unpatched: samples % image name symbol name 6754 83.7861 libz.so.1.2.3 /lib64/libz.so.1.2.3 618 7.6665 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0 534 6.6245 postgres hex_encode 95 1.1785 libc-2.12.so memcpy Patched: samples % image name symbol name 751 50.1670 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0 594 39.6794 postgres hex_encode 83 5.5444 libc-2.12.so memcpy (the test case is selecting one 27 MB bytea in text mode over a localhost connection) Are there any objections to this? It is possible that this could cause a performance regression for people who SELECT lots of compressible data over really slow network connections, but is that a realistic scenario? If there are concerns about that, maybe a GUC variable like ssl_compression (defaulting to off) would be a solution. Yours, Laurenz Albe
Attachment
On Tue, Nov 8, 2011 at 14:59, Albe Laurenz <laurenz.albe@wien.gv.at> wrote: > I ran into a performance problem described in this thread: > > http://archives.postgresql.org/pgsql-performance/2011-10/msg00249.php > continued here: > http://archives.postgresql.org/pgsql-performance/2011-11/msg00045.php > > OpenSSL compresses data by default, and that causes a > performance penalty of 100% and more, at least when > SELECTing larger bytea objects. > The backend process becomes CPU bound. > > From OpenSSL version 1.0.0. on, compression can be > disabled. The attached patch does that, and with that > patch I see dramatic performance improvements: > > Unpatched: > > samples % image name symbol name > 6754 83.7861 libz.so.1.2.3 /lib64/libz.so.1.2.3 > 618 7.6665 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0 > 534 6.6245 postgres hex_encode > 95 1.1785 libc-2.12.so memcpy > > Patched: > > samples % image name symbol name > 751 50.1670 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0 > 594 39.6794 postgres hex_encode > 83 5.5444 libc-2.12.so memcpy > > (the test case is selecting one 27 MB bytea in text > mode over a localhost connection) > > Are there any objections to this? This should probably be made an option. And doesn't it, at least in a lot of cases, make more sense to control this from the client side? It might typically be good to use comopression if you are connecting over a slow link such as mobile or satellite. And typically the client knows that, not the server. So either client, or pg_hba driven, perhaps? > It is possible that this could cause a performance > regression for people who SELECT lots of compressible > data over really slow network connections, but is that > a realistic scenario? Turning it off unconditionally can certainly create such a regression. I don't think it's at all unrealstic. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/
On 11/08/2011 03:59 PM, Albe Laurenz wrote: > If there are concerns about that, maybe a GUC variable like > ssl_compression (defaulting to off) would be a solution. > I'd vote for a libpq connect option instead. Something like sslcompress=yes|no accompanied by PGSSLCOMPRESS environment variable. And defaulting to "yes", as not to break any backward compatibilty. For instance we expect SSL to provide compression, wouldn't even use it without it. Regards, Martin
On Tue, Nov 8, 2011 at 3:59 PM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote: > It is possible that this could cause a performance > regression for people who SELECT lots of compressible > data over really slow network connections, but is that > a realistic scenario? Yes, it's a realistic scenario. Please make it a option. Also, high-security links may prefer compression. -- marko
Marko Kreen <markokr@gmail.com> writes:
> On Tue, Nov 8, 2011 at 3:59 PM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:
>> It is possible that this could cause a performance
>> regression for people who SELECT lots of compressible
>> data over really slow network connections, but is that
>> a realistic scenario?
> Yes, it's a realistic scenario. Please make it a option.
I distinctly recall us getting bashed a few years ago because there
wasn't any convenient way to turn SSL compression *on*. Now that SSL
finally does the sane thing by default, you want to turn it off?
The fact of the matter is that in most situations where you want SSL,
ie links across insecure WANs, compression is a win. Testing a local
connection, as you seem to have done, is just about 100% irrelevant to
performance in the real world.
There might be some argument for providing a client option to disable
compression, but it should not be forced, and it shouldn't even be the
default. But before adding YA connection option, I'd want to see some
evidence that it's useful over non-local connections.
regards, tom lane
On Tue, Nov 8, 2011 at 2:34 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > The fact of the matter is that in most situations where you want SSL, > ie links across insecure WANs, compression is a win. Testing a local > connection, as you seem to have done, is just about 100% irrelevant to > performance in the real world. I would disagree with that. Deployments in the cloud may have fast, but untrustworthy network connections. -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 11/08/2011 09:34 AM, Tom Lane wrote: > Marko Kreen<markokr@gmail.com> writes: >> On Tue, Nov 8, 2011 at 3:59 PM, Albe Laurenz<laurenz.albe@wien.gv.at> wrote: >>> It is possible that this could cause a performance >>> regression for people who SELECT lots of compressible >>> data over really slow network connections, but is that >>> a realistic scenario? >> Yes, it's a realistic scenario. Please make it a option. > I distinctly recall us getting bashed a few years ago because there > wasn't any convenient way to turn SSL compression *on*. Now that SSL > finally does the sane thing by default, you want to turn it off? > > The fact of the matter is that in most situations where you want SSL, > ie links across insecure WANs, compression is a win. Testing a local > connection, as you seem to have done, is just about 100% irrelevant to > performance in the real world. > > There might be some argument for providing a client option to disable > compression, but it should not be forced, and it shouldn't even be the > default. But before adding YA connection option, I'd want to see some > evidence that it's useful over non-local connections. I can certainly conceive of situations where one wants SSL on a high speed/bandwidth network. I don't think we should assume that all or even most real world SSL use will be across slow networks. Here's another data point: <http://journal.paul.querna.org/articles/2011/04/05/openssl-memory-use/> cheers andrew
Andrew Dunstan <andrew@dunslane.net> writes:
> On 11/08/2011 09:34 AM, Tom Lane wrote:
>> There might be some argument for providing a client option to disable
>> compression, but it should not be forced, and it shouldn't even be the
>> default. But before adding YA connection option, I'd want to see some
>> evidence that it's useful over non-local connections.
> I can certainly conceive of situations where one wants SSL on a high
> speed/bandwidth network. I don't think we should assume that all or even
> most real world SSL use will be across slow networks.
Even for that use-case, I don't believe that testing on a local loopback
connection should be considered representative.
regards, tom lane
On Tue, Nov 8, 2011 at 9:58 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Andrew Dunstan <andrew@dunslane.net> writes: >> On 11/08/2011 09:34 AM, Tom Lane wrote: >>> There might be some argument for providing a client option to disable >>> compression, but it should not be forced, and it shouldn't even be the >>> default. But before adding YA connection option, I'd want to see some >>> evidence that it's useful over non-local connections. > >> I can certainly conceive of situations where one wants SSL on a high >> speed/bandwidth network. I don't think we should assume that all or even >> most real world SSL use will be across slow networks. > > Even for that use-case, I don't believe that testing on a local loopback > connection should be considered representative. Probably not, but I think we ought to provide the option to disable compression for those who want to do that. I also agree with you that we should leave the default as-is. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Tue, Nov 8, 2011 at 4:34 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Marko Kreen <markokr@gmail.com> writes: >> On Tue, Nov 8, 2011 at 3:59 PM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote: >>> It is possible that this could cause a performance >>> regression for people who SELECT lots of compressible >>> data over really slow network connections, but is that >>> a realistic scenario? > >> Yes, it's a realistic scenario. Please make it a option. > > I distinctly recall us getting bashed a few years ago because there > wasn't any convenient way to turn SSL compression *on*. Now that SSL > finally does the sane thing by default, you want to turn it off? > > The fact of the matter is that in most situations where you want SSL, > ie links across insecure WANs, compression is a win. Testing a local > connection, as you seem to have done, is just about 100% irrelevant to > performance in the real world. > > There might be some argument for providing a client option to disable > compression, but it should not be forced, and it shouldn't even be the > default. But before adding YA connection option, I'd want to see some > evidence that it's useful over non-local connections. +1 for keeping current default. But I can imagine scenarios where having option to turn compression off could be useful: - when minimal latency is required - when "normal" latency is required, but data is big - when serving big non-compressible blobs - zlib can be very slow - when serving lots of connections and want to minimize unnecessary cpu and memory load Depending on how zlib is used by openssl, some of them may not happen in practice. -- marko
Tom Lane wrote: > I distinctly recall us getting bashed a few years ago because there > wasn't any convenient way to turn SSL compression *on*. Now that SSL > finally does the sane thing by default, you want to turn it off? > > The fact of the matter is that in most situations where you want SSL, > ie links across insecure WANs, compression is a win. Testing a local > connection, as you seem to have done, is just about 100% irrelevant to > performance in the real world. Maybe that's paranoia, but we use SSL via the company's LAN to keep potentially sensitive data from crossing the network unencrypted. > There might be some argument for providing a client option to disable > compression, but it should not be forced, and it shouldn't even be the > default. But before adding YA connection option, I'd want to see some > evidence that it's useful over non-local connections. I will try to provide test results via remote connection; I thought that localhost was a good enough simulation for a situation where you are not network bound. I agree with you that a client option would make more sense. The big problem I personally have with that is that it only works if you use libpq. When using the JDBC driver or Npgsql, a client option wouldn't help me at all. Yours, Laurenz Albe
Tom Lane wrote: > There might be some argument for providing a client option to disable > compression, but it should not be forced, and it shouldn't even be the > default. But before adding YA connection option, I'd want to see some > evidence that it's useful over non-local connections. Here are numbers from a test via LAN. The client machine has OpenSSL 0.9.8e, the server OpenSSL 1.0.0. The client command run was echo 'select ...' | time psql "host=..." -o /dev/null and \timing was turned on in .psqlrc In addition to the oprofile data I collected three times: - the duration as shown in the server log - the duration as shown by \timing - the duration of the psql command as measured by "time" Without patch: duration: 5730.996 ms (log), 5975.093 ms (\timing), 22.87 s (time) samples % image name symbol name 4428 80.2029 libz.so.1.2.3 /lib64/libz.so.1.2.3 559 10.1250 postgres hex_encode 361 6.5387 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0 83 1.5034 libc-2.12.so memcpy With patch: duration: 3001.009 ms (log), 3243.690 ms (\timing), 20.27 s (time) samples % image name symbol name 1072 58.0401 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0 587 31.7813 postgres hex_encode 105 5.6849 libc-2.12.so memcpy I think this makes a good case for disabling compression. Yours, Laurenz Albe
On Tue, Nov 08, 2011 at 04:59:02PM +0100, Albe Laurenz wrote: > Tom Lane wrote: > > There might be some argument for providing a client option to disable > > compression, but it should not be forced, and it shouldn't even be the > > default. But before adding YA connection option, I'd want to see some > > evidence that it's useful over non-local connections. > > Here are numbers from a test via LAN. > The client machine has OpenSSL 0.9.8e, the server OpenSSL 1.0.0. > > The client command run was > > echo 'select ...' | time psql "host=..." -o /dev/null > > and \timing was turned on in .psqlrc > > In addition to the oprofile data I collected three times: > - the duration as shown in the server log > - the duration as shown by \timing > - the duration of the psql command as measured by "time" > > Without patch: > > duration: 5730.996 ms (log), 5975.093 ms (\timing), 22.87 s (time) > > samples % image name symbol name > 4428 80.2029 libz.so.1.2.3 /lib64/libz.so.1.2.3 > 559 10.1250 postgres hex_encode > 361 6.5387 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0 > 83 1.5034 libc-2.12.so memcpy > > With patch: > > duration: 3001.009 ms (log), 3243.690 ms (\timing), 20.27 s (time) > > samples % image name symbol name > 1072 58.0401 libcrypto.so.1.0.0 /usr/lib64/libcrypto.so.1.0.0 > 587 31.7813 postgres hex_encode > 105 5.6849 libc-2.12.so memcpy > > > I think this makes a good case for disabling compression. > > Yours, > Laurenz Albe Certainly a good case for providing the option to disable compression. Regards, Ken
On Tue, Nov 08, 2011 at 04:19:02PM +0100, Albe Laurenz wrote: > Tom Lane wrote: > > I distinctly recall us getting bashed a few years ago because there > > wasn't any convenient way to turn SSL compression *on*. Now that SSL > > finally does the sane thing by default, you want to turn it off? > > > > The fact of the matter is that in most situations where you want SSL, > > ie links across insecure WANs, compression is a win. Testing a local > > connection, as you seem to have done, is just about 100% irrelevant to > > performance in the real world. > > Maybe that's paranoia, but we use SSL via the company's LAN to keep > potentially sensitive data from crossing the network unencrypted. > > > There might be some argument for providing a client option to disable > > compression, but it should not be forced, and it shouldn't even be the > > default. But before adding YA connection option, I'd want to see some > > evidence that it's useful over non-local connections. > > I will try to provide test results via remote connection; I thought > that localhost was a good enough simulation for a situation where > you are not network bound. > > I agree with you that a client option would make more sense. > The big problem I personally have with that is that it only works > if you use libpq. When using the JDBC driver or Npgsql, a client > option wouldn't help me at all. > > Yours, > Laurenz Albe > I think that JDBC and Npgsql should also support disabling compression. Regards, Ken
On Tue, Nov 8, 2011 at 11:06 AM, ktm@rice.edu <ktm@rice.edu> wrote: > I think that JDBC and Npgsql should also support disabling compression. That's the *real* problem here... You're quite right that if we allow controlling this on the libpq side, it is surely desirable to allow controlling this via JDBC, Npgsql, and other mechanisms people may have around. (There are native protocol implementations for Common Lisp and Go, for instance. They may not be particularly important, ) Unfortunately, each protocol implementation is independent, which really is the nature of the beast, which includes: a) The code of the implementation, b) Release of the implementation, c) Packaging of releases into software distributions. With that series of complications, I wonder if maybe the right place to control this is pg_hba.conf. That has the merit of centralizing control in such a way that it would apply commonly to libpq/JDBC/Npgsql/..., though with the demerit that the control does not take place on the client side, which is desirable. I wonder how many SSL parameters there are which would be worth trying to have available. I expect we'd benefit from looking at all the relevant ones at once, so as to not have the problem of hacking "one more" into place and perhaps doing it a bit differently each time. -- When confronted by a difficult problem, solve it by reducing it to the question, "How would the Lone Ranger handle this?"
On 2011-11-08 22:59, Albe Laurenz wrote: > In addition to the oprofile data I collected three times: > - the duration as shown in the server log > - the duration as shown by \timing > - the duration of the psql command as measured by "time" [...] > I think this makes a good case for disabling compression. It's a few data points, but is it enough to make a good case? As I understand it, compression can save time not only on transport but also on the amount of data that needs to go through encryption -- probably depending on choice of cypher, hardware support, machine word width, compilation details etc. Would it make sense to run a wider experiment, e.g. in the buld farm? Another reason why I believe compression is often used with encryption is to maximize information content per byte of data: harder to guess, harder to crack. Would that matter? Jeroen
Jeroen Vermeulen <jtv@xs4all.nl> writes:
> Another reason why I believe compression is often used with encryption
> is to maximize information content per byte of data: harder to guess,
> harder to crack. Would that matter?
Yes, it would. There's a reason why the OpenSSL default is what it is.
regards, tom lane
Tom,
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> The fact of the matter is that in most situations where you want SSL,
> ie links across insecure WANs, compression is a win. Testing a local
> connection, as you seem to have done, is just about 100% irrelevant to
> performance in the real world.
I'm mystified by the idea that SSL shouldn't be used on local networks.
If the only things talking to the database are other servers on
physically secure networks, perhaps, but when you've got databases
exposed (even through firewalls) to client networks (which are in the
same building), and any data that's even remotely sensetive, you should
be using SSL or IPSEC. The chances of eaves-dropping on a typiacal WAN
physical/dedicated link (not over the Internet..) are actually much less
than some disgruntled employee spoofing the local switches to monitor
someone else's traffic. For starters, you're going to need some pretty
specialized gear to eavesdrop on a T1 or similar link and once it's past
the last mile and into the fibre network... Well, there's some folks
who can manage that, but it's not very many.
> There might be some argument for providing a client option to disable
> compression, but it should not be forced, and it shouldn't even be the
> default. But before adding YA connection option, I'd want to see some
> evidence that it's useful over non-local connections.
I agree that it should be an option and that it should be on by default.
It's going to typically be a win.
Thanks,
Stephen
Christopher Browne wrote: >> I think that JDBC and Npgsql should also support disabling compression. > > That's the *real* problem here... > > You're quite right that if we allow controlling this on the libpq > side, it is surely desirable to allow controlling this via JDBC, > Npgsql, and other mechanisms people may have around. [...] > With that series of complications, I wonder if maybe the right place > to control this is pg_hba.conf. I think that wouldn't work, because to query pg_hba.conf, you have to know user and database, which come from the client side. But the SSL negotiation takes place earlier, namely when the connection is established. > I wonder how many SSL parameters there are which would be worth trying > to have available. I expect we'd benefit from looking at all the > relevant ones at once, so as to not have the problem of hacking "one > more" into place and perhaps doing it a bit differently each time. Sure, if anybody can think of any. A quick look at "man SSL_CTX_set_options" didn't show me any, but then OpenSSL's documentation is very bad (the page does not even mention SSL_OP_NO_COMPRESSION) and I am no SSL expert. Is the following proposal acceptable: - Add a GUC ssl_compression, defaulting to "on". - Add a client option "sslcompression" and an environment variable PGSSLCOMPRESSION, defaulting to "1". Compression will be disabled if either side refuses. That way you can control the setting per client, but you can also force it on clients that do not use libpq if you want. Yours, Laurenz Albe
On Wednesday, November 9, 2011, Albe Laurenz wrote:
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
Christopher Browne wrote:
>> I think that JDBC and Npgsql should also support disabling
compression.
>
> That's the *real* problem here...
>
> You're quite right that if we allow controlling this on the libpq
> side, it is surely desirable to allow controlling this via JDBC,
> Npgsql, and other mechanisms people may have around.
[...]
> With that series of complications, I wonder if maybe the right place
> to control this is pg_hba.conf.
I think that wouldn't work, because to query pg_hba.conf, you have to
know user and database, which come from the client side.
But the SSL negotiation takes place earlier, namely when the
connection is established.
Oh, right, that's going to be a problem doing it there.
> I wonder how many SSL parameters there are which would be worth trying
> to have available. I expect we'd benefit from looking at all the
> relevant ones at once, so as to not have the problem of hacking "one
> more" into place and perhaps doing it a bit differently each time.
Sure, if anybody can think of any. A quick look at
"man SSL_CTX_set_options" didn't show me any, but then OpenSSL's
documentation is very bad (the page does not even mention
SSL_OP_NO_COMPRESSION) and I am no SSL expert.
Oh yeah, their docs are. Um. Yeah, let's leave it at that.
I think the other one is to control which encryption options are available - but we already have a guc for that.
Is the following proposal acceptable:
- Add a GUC ssl_compression, defaulting to "on".
- Add a client option "sslcompression" and an environment variable
PGSSLCOMPRESSION, defaulting to "1".
Seems like the reasonable thing, yes.
Compression will be disabled if either side refuses.
I assume OpenSSL takes care of this for us, right? We just have to set the flags on the connection?
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
Magnus Hagander <magnus@hagander.net> writes:
> On Wednesday, November 9, 2011, Albe Laurenz wrote:
>> Is the following proposal acceptable:
>>
>> - Add a GUC ssl_compression, defaulting to "on".
>> - Add a client option "sslcompression" and an environment variable
>> PGSSLCOMPRESSION, defaulting to "1".
> Seems like the reasonable thing, yes.
A GUC is entirely, completely, 100% the wrong answer. It has no way to
deal with the fact that some clients may need compression and others
not.
It should be a client option, full stop. The fact that that will be
more work to implement does not make "kluge it at the server" the right
answer.
regards, tom lane
On Wed, Nov 09, 2011 at 05:26:14PM -0500, Tom Lane wrote: > Magnus Hagander <magnus@hagander.net> writes: > > On Wednesday, November 9, 2011, Albe Laurenz wrote: > >> Is the following proposal acceptable: > >> > >> - Add a GUC ssl_compression, defaulting to "on". > >> - Add a client option "sslcompression" and an environment variable > >> PGSSLCOMPRESSION, defaulting to "1". > > > Seems like the reasonable thing, yes. > > A GUC is entirely, completely, 100% the wrong answer. It has no way to > deal with the fact that some clients may need compression and others > not. > > It should be a client option, full stop. The fact that that will be > more work to implement does not make "kluge it at the server" the right > answer. Assuming the GUC is PGC_BACKEND or better, what is the distinction beyond the cosmetic "sslcompression=off" vs. "options='-c ssl_compression=off'"? Does OpenSSL respect changes to this setting during a connection's lifetime? If so, we could offer a PGC_USERSET GUC, enabling the client to even alternate use of compression within a single session.
Magnus Hagander wrote: >> Is the following proposal acceptable: >> >> - Add a GUC ssl_compression, defaulting to "on". >> - Add a client option "sslcompression" and an environment variable >> PGSSLCOMPRESSION, defaulting to "1". > > Seems like the reasonable thing, yes. > >> Compression will be disabled if either side refuses. > > I assume OpenSSL takes care of this for us, right? We just have to set the flags on the connection? Right. Yours, Laurenz Albe
On 11/08/2011 12:39 PM, Tom Lane wrote: > Jeroen Vermeulen<jtv@xs4all.nl> writes: >> Another reason why I believe compression is often used with encryption >> is to maximize information content per byte of data: harder to guess, >> harder to crack. Would that matter? > Yes, it would. There's a reason why the OpenSSL default is what it is. > > An interesting data point on this is that RedHat's nss_compat_ossl package doesn't support SSL compression at all <http://fedoraproject.org/wiki/Nss_compat_ossl>, and it's supposed to be a path to FIPS 140 compliance: <http://fedoraproject.org/wiki/FedoraCryptoConsolidation>. The latter URL, incidentally, contains a lot of good information, and lays out many of the reasons why I'd like to see us support NSS as an alternative to OpenSSL, notwithstanding the supposed dirtiness of its API. I imagine this would be of interest to commercial Postgres vendors also. cheers andrew
On Thursday, November 10, 2011, Andrew Dunstan wrote:
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
On 11/08/2011 12:39 PM, Tom Lane wrote:Jeroen Vermeulen<jtv@xs4all.nl> writes:Another reason why I believe compression is often used with encryptionYes, it would. There's a reason why the OpenSSL default is what it is.
is to maximize information content per byte of data: harder to guess,
harder to crack. Would that matter?
An interesting data point on this is that RedHat's nss_compat_ossl package doesn't support SSL compression at all <http://fedoraproject.org/wiki/Nss_compat_ossl>, and it's supposed to be a path to FIPS 140 compliance: <http://fedoraproject.org/wiki/FedoraCryptoConsolidation>. The latter URL, incidentally, contains a lot of good information, and lays out many of the reasons why I'd like to see us support NSS as an alternative to OpenSSL, notwithstanding the supposed dirtiness of its API. I imagine this would be of interest to commercial Postgres vendors also.
Interesting points. I hadn't really considered it from the FIPS perspective.
I thought the main idea was that if we want to support another one it's probably going to be GnuTLS because that one offers key-file-compatibility with OpenSSL, which NSS doesnät.
//Magnus
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
Tom Lane wrote: >>> Is the following proposal acceptable: >>> >>> - Add a GUC ssl_compression, defaulting to "on". >>> - Add a client option "sslcompression" and an environment variable >>> PGSSLCOMPRESSION, defaulting to "1". > A GUC is entirely, completely, 100% the wrong answer. It has no way to > deal with the fact that some clients may need compression and others > not. If you leave the GUC at its default value, you can control compression on the client side. You can force a certain SSL cipher on the client, why not a compression setting? > It should be a client option, full stop. The fact that that will be > more work to implement does not make "kluge it at the server" the right > answer. I could go and try to convince Npgsql and JDBC to accept patches to do that on the client side, but that would be more effort than I want to invest. But then there's still closed source software like Devart dotConnect... In my environment it would make sense to control the setting on the server side, because all our database clients connect via LAN, and network bandwidth is not the bottleneck in our database applications. Yours, Laurenz Albe
"Albe Laurenz" <laurenz.albe@wien.gv.at> writes:
> Tom Lane wrote:
>> A GUC is entirely, completely, 100% the wrong answer. It has no way
>> to deal with the fact that some clients may need compression and others
>> not.
> You can force a certain SSL cipher on the client, why not a compression
> setting?
To my mind, the argument for the ssl_cipher setting is to allow the DBA
to enforce a site-wide security policy to the effect of "we consider
only these ciphers strong enough for production use". It's a pretty
weak argument (especially since the setting is not cognizant of where
the connection is coming from), but at least it's an argument.
There's no comparable security argument for an ssl_compression setting.
If anything, a security-minded DBA might wish to insist on compression
being *on*, but you aren't proposing to give him control in that
direction (and AFAICT openssl doesn't offer a force-on flag for it).
But in any case, my objection is that there's no adequate use-case
for this GUC, because it's much more sensible to set it from the client
side. We have too many GUCs already --- Josh B regularly goes on the
warpath looking for ones we can remove. This one should never get in
there to start with.
> I could go and try to convince Npgsql and JDBC to accept patches to
> do that on the client side, but that would be more effort than I
> want to invest. But then there's still closed source software like
> Devart dotConnect...
This argument reads as nothing except "I'm too lazy to solve it right,
so I want you to accept a wrong solution".
regards, tom lane
On Thu, Nov 10, 2011 at 11:01 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > But in any case, my objection is that there's no adequate use-case > for this GUC, because it's much more sensible to set it from the client > side. We have too many GUCs already --- Josh B regularly goes on the > warpath looking for ones we can remove. This one should never get in > there to start with. Of course, we also have no shortage of connection parameters. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Thu, Nov 10, 2011 at 5:18 PM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote: > I could go and try to convince Npgsql and JDBC to accept patches to > do that on the client side, but that would be more effort than I > want to invest. But then there's still closed source software like > Devart dotConnect... Are you certain Java/C# even support SSL compression? Quick grep over some old jdk code I had around did not find it... In any case, the connection libraries do not need to match such optional features. -- marko
Tom Lane wrote: > "Albe Laurenz" <laurenz.albe@wien.gv.at> writes: > > Tom Lane wrote: > >> A GUC is entirely, completely, 100% the wrong answer. It has no way > >> to deal with the fact that some clients may need compression and others > >> not. > > > You can force a certain SSL cipher on the client, why not a compression > > setting? > > To my mind, the argument for the ssl_cipher setting is to allow the DBA > to enforce a site-wide security policy to the effect of "we consider > only these ciphers strong enough for production use". It's a pretty > weak argument (especially since the setting is not cognizant of where > the connection is coming from), but at least it's an argument. > > There's no comparable security argument for an ssl_compression setting. > If anything, a security-minded DBA might wish to insist on compression > being *on*, but you aren't proposing to give him control in that > direction (and AFAICT openssl doesn't offer a force-on flag for it). > > But in any case, my objection is that there's no adequate use-case > for this GUC, because it's much more sensible to set it from the client > side. We have too many GUCs already --- Josh B regularly goes on the > warpath looking for ones we can remove. This one should never get in > there to start with. How is the compression connection parameter set? It seems odd for it to be compiled into the application because the application could be run on different networks. I don't know of any way to inject connection options from outside the application like libpq's PGOPTIONS. Would we add a libpq environment variable for this, like PGUSER? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Bruce Momjian <bruce@momjian.us> writes:
> How is the compression connection parameter set? It seems odd for it to
> be compiled into the application because the application could be run on
> different networks. I don't know of any way to inject connection
> options from outside the application like libpq's PGOPTIONS.
Huh? You put it in the connection string, typically. This is not
different from how you'd specify sslmode to start with.
regards, tom lane
Tom Lane wrote: > Bruce Momjian <bruce@momjian.us> writes: > > How is the compression connection parameter set? It seems odd for it to > > be compiled into the application because the application could be run on > > different networks. I don't know of any way to inject connection > > options from outside the application like libpq's PGOPTIONS. > > Huh? You put it in the connection string, typically. This is not > different from how you'd specify sslmode to start with. Well, you are saying the client is more flexible, but if the client is a binary, it isn't flexible without an environment variable to control it. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Bruce Momjian <bruce@momjian.us> writes:
> Tom Lane wrote:
>> Huh? You put it in the connection string, typically. This is not
>> different from how you'd specify sslmode to start with.
> Well, you are saying the client is more flexible, but if the client is a
> binary, it isn't flexible without an environment variable to control it.
As long as the client can take a connection string, it's perfectly
flexible. If it can't, this is just one more reason why it should
be fixed to do so.
regards, tom lane
Tom Lane wrote: > But in any case, my objection is that there's no adequate use-case > for this GUC, because it's much more sensible to set it from the client > side. We have too many GUCs already --- Josh B regularly goes on the > warpath looking for ones we can remove. This one should never get in > there to start with. I agree that it is sensible to have the setting on the client, and that there should not be too many GUCs. >> I could go and try to convince Npgsql and JDBC to accept patches to >> do that on the client side, but that would be more effort than I >> want to invest. But then there's still closed source software like >> Devart dotConnect... > > This argument reads as nothing except "I'm too lazy to solve it right, > so I want you to accept a wrong solution". In a way, yes, except that I think that "wrong" is exaggerated. As DBA I like to have an option to control things from the server end -- if that's laziness, so be it. So, should I forget about the GUC or is anybody going to back me? I'd still be willing to write a patch for a client-only solution. Yours, Laurenz Albe
Albe Laurenz wrote: > Tom Lane wrote: > > But in any case, my objection is that there's no adequate use-case > > for this GUC, because it's much more sensible to set it from the > client > > side. We have too many GUCs already --- Josh B regularly goes on the > > warpath looking for ones we can remove. This one should never get in > > there to start with. > > I agree that it is sensible to have the setting on the client, > and that there should not be too many GUCs. > > >> I could go and try to convince Npgsql and JDBC to accept patches to > >> do that on the client side, but that would be more effort than I > >> want to invest. But then there's still closed source software like > >> Devart dotConnect... > > > > This argument reads as nothing except "I'm too lazy to solve it right, > > so I want you to accept a wrong solution". > > In a way, yes, except that I think that "wrong" is exaggerated. > As DBA I like to have an option to control things from the server > end -- if that's laziness, so be it. > > So, should I forget about the GUC or is anybody going to back me? > > I'd still be willing to write a patch for a client-only solution. Agreed. There is clearly a win in turning off SSL compression for certain workloads, and if people think the client is the right location, then let's do it there. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Bruce Momjian wrote: >> I'd still be willing to write a patch for a client-only solution. > > Agreed. There is clearly a win in turning off SSL compression for > certain workloads, and if people think the client is the right location, > then let's do it there. Here it is. I'll add it to the November commitfest. Yours, Laurenz Albe
Attachment
I wrote: > Here it is. I'll add it to the November commitfest. Here is the second version. I realized that it is better to set the option on the SSL object and not on the SSL context so that it is possible to change it per connection. I also improved the documentation. Yours, Laurenz Albe
Attachment
On Thu, Nov 17, 2011 at 10:11, Albe Laurenz <laurenz.albe@wien.gv.at> wrote: > I wrote: >> Here it is. I'll add it to the November commitfest. > > Here is the second version. > > I realized that it is better to set the option on the SSL object > and not on the SSL context so that it is possible to change it > per connection. > > I also improved the documentation. Thanks, applied. I changed the documentation around a bit further - I think it's easier to read now. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/