Re: [Proposal] Table-level Transparent Data Encryption (TDE) and KeyManagement Service (KMS) - Mailing list pgsql-hackers

Generally, I think our interest should be less in how secrets are
stored inside the database than in how we can integrate with an
external secrets-management solution, and I think that's what Jeremy
is talking about here.  I don't know exactly how that would work, but
you can imagine having a way to tell an FDW "hey, there's a password
for this server, but it's not stored here -- instead go fetch secret
d41d8cd98f00b204e9800998ecf8427e" and the server does that and uses
that password for the connection.  But we don't need to solve the FDW
problem for this effort to move forward.  We do, however, need a
solution that's good enough for whatever we want to do in terms of
TDE.

Right: the biggest use case I'm thinking about is external secret management systems. For anyone with heavy-weight security requirements, this will be a must.  I use LastPass in my personal life and they have an enterprise product with API access [1] that I know nothing about.  :)  At one previous company where I worked, they used Thycotic [2] which is now OEMed as IBM Security Server [3]. HashiCorp Vault [4] is pretty widely known, and it's docs have a handy list of a whole bunch more Secrets Engines [5] they integrate with.  Every major cloud provider has a secrets solution (AWS [6], Azure [7], GCP [8], etc) and then there are open source secret management suites like Cerberus [9] that layer on top of cloud APIs.  There are the services built into orchestration frameworks like Docker [10] and k8s [11].  And of course, don't forget HSMs [12].

I'm in no way discrediting a full implementation within an extension as you've done, Bruce, with pgcryptokey.  In fact I think we will need something like this as a reference implementation, and to build unit tests.  But the problem is that this doesn't provide a standard API for extensions to code against.  Other extensions need a dependency on pgcryptokey, it's up to each extension author to support every secret provider, and realistically FDWs and logical rep can't ever use an API that's not in core.  In my ideal world, core gives us a standard API that internal code and extensions can each code against to (1) store/retrieve secrets [including temporary secrets or tokens; supporting models like kerberos] and (2) provide custom backend implementations to that service.

In short, core could provide the _plumbing_ of a standard secrets API and allow extensions to register as providers and act as consumers of the API.  FDWs, logical replication and TDE are the things on the table right now but there are lots of conceivable things that future developers might need secrets for.  TDE is a great excuse to get an API in place. If it proves successful, then later on we can look at updating logical replication and FDWs to use this API as well.

-Jeremy


1. https://www.lastpass.com/enterprise-password-management
2. https://thycotic.com/products/secret-server/
3. https://ovum.informa.com/resources/product-content/ibm-adopts-thycotic-for-privileged-account-management
4. https://www.vaultproject.io/
5. https://www.vaultproject.io/docs/secrets/index.html
6. https://aws.amazon.com/secrets-manager/
7. https://azure.microsoft.com/en-us/services/key-vault/
8. https://cloud.google.com/solutions/secrets-management/
9. http://engineering.nike.com/cerberus/docs/
10. https://docs.docker.com/engine/swarm/secrets/
11. https://kubernetes.io/docs/concepts/configuration/secret/
12. https://en.wikipedia.org/wiki/Hardware_security_module

-- 
Jeremy Schneider
Database Engineer
Amazon Web Services