Thread: WIP: Secure Transport support as OpenSSL alternative on macOS
As noted in 77FB2321-9210-4AD4-B7FC-67623A5838F0@justatheory.com, OpenSSL is being somewhat dismantled on macOS starting with 10.11 El Capitan. To scratch my own itch I decided to take a stab at adding support for the Secure Transport library. The idea is to make it as much of a drop-in for the OpenSSL support as possible, supporting the same file formats etc (Secure Transport have PKCS12 API calls while support PEM requires using lower level APIs). Only the frontend has been implemented so far. I used a vanilla OpenSSL- compiled backend for testing (see patch 0002 below). Currently the CRL tests fail since Secure Transport doesn’t handle manual CRL lists like OpenSSL, CRL handling is done automatically by the Keychain upon validation. One alternative for supporting this could perhaps be to create a temporary Keychain for the duration of the connection over which more control can be had (need to research this further). For now, no Secure Transport specifics are supported but the idea is to add support for loading certificates and keys from the Keychain as well as from the usual locations. The engine and compression settings will however be no-ops since there is no support in Secure Transport for these. Being an early WIP show-and-tell submissions and not something up for code review (anyone wanting to take a look is of course more than welcome), it is deliberately rough around the edges. Except the manual CRL loading, it’s mostly featurecomplete with the OpenSSL frontend, TODO items include: removing use of private and deprecated API calls; ensure release of all resources; support certificates/keys in Keychain; possibly use temporary keychains for connections; support passphrase entering (there has been lots of deprecations around that, need to figure out the supported way forward); tidy up, and write better, comments; documentation. The main questions raised here are: is it of interest to support multiple SSL libraries given the additional support burden and; is supporting Secure Transport of any interest or is it a wasted effort to continue towards a full frontend/backend/doc submission? On top of the OpenSSL situation in macOS, supporting Secure Transport makes sense if the Keychain can be fully leveraged since that would allow IT organisations to centrally manage and provision Mac clients without passing certificate/key files around. Should there be any interest in this I intend to continue on one more library which is more geared towards server environments like nss or <suggestions-welcome> once Secure Transport is done (scratching another itch). The attached show-and-tell patches are: 0001. The WIP frontend support for Secure Transport together with minimal scaffolding around it such as Makefile and autoconf. 0002. Adding support to PostgresNode.pm to run the server binaries from a defined path making it easier to test a frontend without having to a) implement the backend at the same time or b) fiddle with paths manually. The SSL tests are the only ones modified. There is nothing Secure Transport specific in this patch so iff the approach is deemed interesting it could be considered out of band. Since the only consumer is testing SSL during development it might be too invasive however. 0003. A first little stab at tweaking the docs to mention support for multiple SSL libraries. There is a lot left here but the idea is to contain the library specifics in subsections with the main sections being generic SSL support information. On top of tweaking the existing sections I intend (once done) to write up the steps needed to implement support for an SSL library, this should perhaps be a README in the tree though? Thoughts, comments? cheers ./daniel
Attachment
Daniel, * Daniel Gustafsson (daniel@yesql.se) wrote: > The main questions raised here are: is it of interest to support multiple SSL > libraries given the additional support burden and; is supporting Secure > Transport of any interest or is it a wasted effort to continue towards a full > frontend/backend/doc submission? We've already started working towards being able to support multiple libraries (in particular, Heikki's work to try and contain the OpenSSL bits) and I've been looking at what it'll take to add NSS support. Generally speaking, I think we'd be better off looking at adding support for multi-platform libraries like NSS rather than OS-specific APIs, but I do understand how there can be advantages to using the OS API (keychain integration is certainly a big part of that, as you mention; the same is true for our Kerberos/GSS support). > On top of the OpenSSL situation in macOS, supporting Secure Transport makes > sense if the Keychain can be fully leveraged since that would allow IT > organisations to centrally manage and provision Mac clients without passing > certificate/key files around. Should there be any interest in this I intend to > continue on one more library which is more geared towards server environments > like nss or <suggestions-welcome> once Secure Transport is done (scratching > another itch). I'd certainly be more than happy to help with the NSS side of things, though I'd want to have both client and server support for it. > The attached show-and-tell patches are: Haven't actually looked at the patches yet, but very happy that you're also interested in this. :) > 0003. A first little stab at tweaking the docs to mention support for > multiple SSL libraries. There is a lot left here but the idea is to contain > the library specifics in subsections with the main sections being generic > SSL support information. On top of tweaking the existing sections I intend > (once done) to write up the steps needed to implement support for an SSL > library, this should perhaps be a README in the tree though? Agreed, this would be better as a README rather than in the formal docs, since adding support for another SSL library isn't really user-facing. Thanks! Stephen
> On 05 Oct 2016, at 16:45, Stephen Frost <sfrost@snowman.net> wrote: > > Daniel, > > * Daniel Gustafsson (daniel@yesql.se) wrote: >> The main questions raised here are: is it of interest to support multiple SSL >> libraries given the additional support burden and; is supporting Secure >> Transport of any interest or is it a wasted effort to continue towards a full >> frontend/backend/doc submission? > > We've already started working towards being able to support multiple > libraries (in particular, Heikki's work to try and contain the OpenSSL > bits) and I've been looking at what it'll take to add NSS support. Building on his work it’s quite simple to keep this code contained to its own files which is great. > Generally speaking, I think we'd be better off looking at adding support > for multi-platform libraries like NSS rather than OS-specific APIs, but > I do understand how there can be advantages to using the OS API > (keychain integration is certainly a big part of that, as you mention; > the same is true for our Kerberos/GSS support). In general I agree with you, supporting OS specific APIs on common platforms (for some value of) which doesn’t ship with a useable OpenSSL installation can still be of value I think. >> On top of the OpenSSL situation in macOS, supporting Secure Transport makes >> sense if the Keychain can be fully leveraged since that would allow IT >> organisations to centrally manage and provision Mac clients without passing >> certificate/key files around. Should there be any interest in this I intend to >> continue on one more library which is more geared towards server environments >> like nss or <suggestions-welcome> once Secure Transport is done (scratching >> another itch). > > I'd certainly be more than happy to help with the NSS side of things, > though I'd want to have both client and server support for it. I think any library supported should be required to have both client and server support. I stopped short after the frontend in this patch to get feedback but will absolutely implement the backend as well. NSS is on my list and I intend to get started on that somewhat shortly. >> The attached show-and-tell patches are: > > Haven't actually looked at the patches yet, but very happy that you're > also interested in this. :) > >> 0003. A first little stab at tweaking the docs to mention support for >> multiple SSL libraries. There is a lot left here but the idea is to contain >> the library specifics in subsections with the main sections being generic >> SSL support information. On top of tweaking the existing sections I intend >> (once done) to write up the steps needed to implement support for an SSL >> library, this should perhaps be a README in the tree though? > > Agreed, this would be better as a README rather than in the formal docs, > since adding support for another SSL library isn't really user-facing. cheers ./daniel
On Wed, Oct 5, 2016 at 5:36 AM, Daniel Gustafsson <daniel@yesql.se> wrote: > The main questions raised here are: is it of interest to support multiple SSL > libraries given the additional support burden and; is supporting Secure > Transport of any interest or is it a wasted effort to continue towards a full > frontend/backend/doc submission? I think this is highly worthwhile. I wish we could interest someone in doing the work for Windows ... but I'm a macOS user myself, so I'll be happy to have you fix my future compile problems for me. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Oct 5, 2016 at 5:36 AM, Daniel Gustafsson <daniel@yesql.se> wrote:
>> The main questions raised here are: is it of interest to support multiple SSL
>> libraries given the additional support burden and; is supporting Secure
>> Transport of any interest or is it a wasted effort to continue towards a full
>> frontend/backend/doc submission?
> I think this is highly worthwhile. I wish we could interest someone
> in doing the work for Windows ... but I'm a macOS user myself, so I'll
> be happy to have you fix my future compile problems for me.
"Future"? Apple isn't even shipping the OpenSSL headers anymore, and
I imagine soon no libraries either. We really have got little choice
on that platform but to do something with Secure Transport. I'm glad
somebody is taking up the task.
regards, tom lane
On Wed, Oct 5, 2016 at 8:42 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
-- Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Oct 5, 2016 at 5:36 AM, Daniel Gustafsson <daniel@yesql.se> wrote:
>> The main questions raised here are: is it of interest to support multiple SSL
>> libraries given the additional support burden and; is supporting Secure
>> Transport of any interest or is it a wasted effort to continue towards a full
>> frontend/backend/doc submission?
> I think this is highly worthwhile. I wish we could interest someone
> in doing the work for Windows ... but I'm a macOS user myself, so I'll
> be happy to have you fix my future compile problems for me.
"Future"? Apple isn't even shipping the OpenSSL headers anymore, and
I imagine soon no libraries either. We really have got little choice
on that platform but to do something with Secure Transport. I'm glad
somebody is taking up the task.
Sure we do. Windows doesn't ship them either, and yet somehow Postgres manages to work just fine there, including with openssl support. There's nothing more magic about MacOS than there is for Windows.
That said, I agree that somebody is picking up the task. I actually think it would be a lot more useful to get Windows SChannel support (because it's *much* more of a PITA to get OpenSSL onto Windows than it is to get it onto macOS) or even moreso NSS (because then every platform could use that, and they have other integrations too). But one important point is that once we have *two* implementations (openssl + macos) then we will know a lot more about the correct places for abstractions etc, and and adding the third one is probably going to be easier (but not easy). But let's make sure we keep in mind there should be more than just these two implementation when defining any external interfaces.