Thread: Using a stock openssl BIO
Hi, Thomas^WA bad person recently nerdsniped me (with the help of an accidental use of an SSL connection in a benchmark leading to poor results) into checking what would be needed to benefit from SSL/TLS hardware acceleration (available with suitable hardware, OS support (linux and freebsd) and openssl 3). One problem turns out to be the custom BIO we use. Which made me look at why we use those. In the backend the first reason is: * Private substitute BIO: this does the sending and receiving using send() and * recv() instead. This is so that we can enable and disable interrupts * just while calling recv(). We cannot have interrupts occurring while * the bulk of OpenSSL runs, because it uses malloc() and possibly other * non-reentrant libc facilities. I think this part has been obsolete for a while now (primarily [1]). These days we always operate the backend sockets in nonblocking mode, and handle blocking at a higher level. Which is where we then can handle interrupts etc correctly (which we couldn't really before, because it still wasn't ok to jump out of openssl). The second part is * We also need to call send() and recv() * directly so it gets passed through the socket/signals layer on Win32. And the not stated need to set/unset pgwin32_noblock around the recv/send calls. I don't think the signal handling stuff is still needed with nonblocking sockets? It seems we could just ensure that there's a pgwin32_poll_signals() somewhere higher up in secure_read/write()? E.g. in ProcessClientReadInterrupt()/ProcessClientWriteInterrupt() or with an explicit call. And the pgwin32_noblock handling could just be done outside the SSL_read/write(). On the client side it looks like things would be a bit harder. The main problem seems to be dealing with SIGPIPE. We could possibly deal with that by moving the handling of that a layer up. That's a thick nest of ugly stuff :(. A problem shared by FE & BE openssl is that openssl internally uses BIO_set_data() inside the BIO routines we reuse. Oops. I hacked around that using BIO_set_callback_arg()/BIO_get_callback_arg(), but that's not particularly pretty either, although it probably is more reliable, since callbacks are intended to be set separately from the BIO implementation. A better approach might be to move the code using per-bio custom data from pqsecure_raw_read() up to pqsecure_read() etc. If we wanted to be able to use TLS acceleration while continuing to use our custom socket routines we'd have to copy a good bit more functionality from openssl into our BIO implementations:(. FWIW, I don't think hardware tls acceleration is a particularly crucial thing for now. Outside of backups it's rare to have the symmetric encryption part of tls be the problem these days thanks, to the AES etc functions in most of the common CPUs. I don't plan to work on this, but Thomas encouraged me to mention this on the list when I mention it to him. Greetings, Andres Freund [1] https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=387da18874afa17156ee3af63766f17efb53c4b9
On Wed, Jul 14, 2021 at 07:17:47PM -0700, Andres Freund wrote: > FWIW, I don't think hardware tls acceleration is a particularly crucial thing > for now. Outside of backups it's rare to have the symmetric encryption part of > tls be the problem these days thanks, to the AES etc functions in most of the > common CPUs. > > I don't plan to work on this, but Thomas encouraged me to mention this on the > list when I mention it to him. So, I am aware of CPU AES acceleration and I assume PG uses that. It is the public key certificate verification part of TLS that we don't use hardware acceleration for, right? -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com If only the physical world exists, free will is an illusion.
Hi, On 2021-07-15 13:59:26 -0400, Bruce Momjian wrote: > On Wed, Jul 14, 2021 at 07:17:47PM -0700, Andres Freund wrote: > > FWIW, I don't think hardware tls acceleration is a particularly crucial thing > > for now. Outside of backups it's rare to have the symmetric encryption part of > > tls be the problem these days thanks, to the AES etc functions in most of the > > common CPUs. > > > > I don't plan to work on this, but Thomas encouraged me to mention this on the > > list when I mention it to him. > > So, I am aware of CPU AES acceleration and I assume PG uses that. Yes, it does so via openssl. But that still happens on the CPU. And what's more, there's a lot of related work in TLS that's fairly expensive (chunking up the data into TLS records etc). Some of the better NICs can do that work in the happy path, so the CPU doesn't have to do encryption nor framing. In some cases that can avoid the to-be-sent data ever being pulled into the CPU caches, but instead it can be DMA directly to the NIC. In PG's case that's particularly interesting when sending out file data in bulk, say in basebackup.c or walsender.c - the data can be sent via sendfile(), so it never goes to userspace. Here's an overview of the kernel TLS / TLS offload https://legacy.netdevconf.info/1.2/papers/ktls.pdf > It is the public key certificate verification part of TLS that we > don't use hardware acceleration for, right? That's true, but separate from what I was talking about. For most database workloads the public key stuff shouldn't be a major piece, because connection establishment shouldn't be that frequent... Greetings, Andres Freund
> On 15 Jul 2021, at 04:17, Andres Freund <andres@anarazel.de> wrote: > Thomas^WA bad person recently nerdsniped me (with the help of an accidental use > of an SSL connection in a benchmark leading to poor results) into checking what > would be needed to benefit from SSL/TLS hardware acceleration (available with > suitable hardware, OS support (linux and freebsd) and openssl 3). Now why does that sounds so familiar.. =) > In the backend the first reason is: > > * Private substitute BIO: this does the sending and receiving using send() and > * recv() instead. This is so that we can enable and disable interrupts > * just while calling recv(). We cannot have interrupts occurring while > * the bulk of OpenSSL runs, because it uses malloc() and possibly other > * non-reentrant libc facilities. > > I think this part has been obsolete for a while now I concur. > The second part is > * We also need to call send() and recv() > * directly so it gets passed through the socket/signals layer on Win32. > > And the not stated need to set/unset pgwin32_noblock around the recv/send > calls. > > I don't think the signal handling stuff is still needed with nonblocking > sockets? It seems we could just ensure that there's a pgwin32_poll_signals() > somewhere higher up in secure_read/write()? E.g. in > ProcessClientReadInterrupt()/ProcessClientWriteInterrupt() or with an explicit > call. > > And the pgwin32_noblock handling could just be done outside the SSL_read/write(). I hadn't yet looked at the pgwin32 parts in detail, but this is along what I was thinking (just more refined). > On the client side it looks like things would be a bit harder. The main problem > seems to be dealing with SIGPIPE. We could possibly deal with that by moving > the handling of that a layer up. That's a thick nest of ugly stuff :(. My initial plan was to keep this for the backend, as the invasiveness of the frontend patch is unlikely to be justified by the returns of the acceleration. > FWIW, I don't think hardware tls acceleration is a particularly crucial thing > for now. Agreed, it will most likely be of limited use to most. It might however make sense to "get in on the ground floor" to be ready in case it's expanded on in kernel+OpenSSL with postgres automatically just reaping the benefits. Either way I was hoping to get to a patch which is close enough to what it would need to look like so we can decide with the facts at hand. > I don't plan to work on this, but Thomas encouraged me to mention this on the > list when I mention it to him. I still have it on my TODO for after the vacation, and hope to reach that part of the list soon. -- Daniel Gustafsson https://vmware.com/
Hi, On 2021-07-22 00:21:16 +0200, Daniel Gustafsson wrote: > > On 15 Jul 2021, at 04:17, Andres Freund <andres@anarazel.de> wrote: > > > Thomas^WA bad person recently nerdsniped me (with the help of an accidental use > > of an SSL connection in a benchmark leading to poor results) into checking what > > would be needed to benefit from SSL/TLS hardware acceleration (available with > > suitable hardware, OS support (linux and freebsd) and openssl 3). > > Now why does that sounds so familiar.. =) :) > > On the client side it looks like things would be a bit harder. The main problem > > seems to be dealing with SIGPIPE. We could possibly deal with that by moving > > the handling of that a layer up. That's a thick nest of ugly stuff :(. > > My initial plan was to keep this for the backend, as the invasiveness of the > frontend patch is unlikely to be justified by the returns of the acceleration. There's two main reasons I'd prefer not to do that: 1) It makes it surprisingly hard to benchmark the single connection TLS throughput, because there's no client that can pull data quick enough. 2) The main case for wanting offload imo is bulk data stuff (basebackup, normal backup). For that you also want to be able to receive the data fast. Outside of situations like that I don't think the gains are likely to be meaningful given AES-NI and other similar cpu level acceleration. > > FWIW, I don't think hardware tls acceleration is a particularly crucial thing > > for now. > > Agreed, it will most likely be of limited use to most. It might however make > sense to "get in on the ground floor" to be ready in case it's expanded on in > kernel+OpenSSL with postgres automatically just reaping the benefits. Either > way I was hoping to get to a patch which is close enough to what it would need > to look like so we can decide with the facts at hand. Yea. I also just think getting rid of the bio stuff is good for maintainability / robustness. Relying on our own socket functions being compatible with the openssl bio doesn't sound very future proof... Especially not combined with our use of the data field, which of course the other bio functions may use (as they do when ktls is enabled!). > > I don't plan to work on this, but Thomas encouraged me to mention this on the > > list when I mention it to him. > > I still have it on my TODO for after the vacation, and hope to reach that part > of the list soon. Cool! Greetings, Andres Freund