Thread: Trust intermediate CA for client certificates

I am trying to configure PostgreSQL 8.4 to trust an intermediate CA for
client certificate validation -- without trusting everything signed by
the root CA (or a different intermediate CA).  Given the following CA
hierarchy, for example, I would like to trust *only* client certificates
signed by the client CA.

            +---------+
            | Root CA |
            +---------+
                /\
               /  \
              /    \
             /      \
            /        \
           /          \
          /            \
         /              \
  +-----------+    +-----------+
  | Server CA |    | Client CA |
  +-----------+    +-----------+

I expected that I could simply use the client CA certificate as
$PGDATA/root.crt, but this does not work; I get an "unknown ca" error.
AFAICT, there is absolutely no way to make PostgreSQL trust a CA that is
not a self-signed root CA.

I can connect successfully if I add the root CA certificate to the
root.crt file, but would effectively trust any certificate signed by
the root CA or any of its subsidiaries, something that I absolutely do
not want.

Am I missing something?

(And yes I have read the documentation, several times over.  It talks
about adding the intermediate CA certificate(s) to the certificate chain
presented by the client, so that the server can complete the chain
between the client certificate and the trusted root CA.  The use case
described above is not discussed, even though it's pretty fundamental to
the PKI trust model.)

Thanks!

--
========================================================================
Ian Pilcher                                         arequipeno@gmail.com
Sometimes there's nothing left to do but crash and burn...or die trying.
========================================================================

Ian Pilcher <arequipeno@gmail.com> writes:
> I am trying to configure PostgreSQL 8.4 to trust an intermediate CA for
> client certificate validation -- without trusting everything signed by
> the root CA (or a different intermediate CA).  Given the following CA
> hierarchy, for example, I would like to trust *only* client certificates
> signed by the client CA.

>             +---------+
>             | Root CA |
>             +---------+
>                 /\
>                /  \
>               /    \
>              /      \
>             /        \
>            /          \
>           /            \
>          /              \
>   +-----------+    +-----------+
>   | Server CA |    | Client CA |
>   +-----------+    +-----------+

> I expected that I could simply use the client CA certificate as
> $PGDATA/root.crt, but this does not work; I get an "unknown ca" error.

Maybe I'm missing something, but I don't see why you'd expect a
different result.  That leaves you with no way to validate the server's
own certificate.

I think it might work to put both the server CA and client CA certs
(but not the root CA cert) into the server's root.crt.

            regards, tom lane

On 03/07/2013 08:28 AM, Tom Lane wrote:
> Maybe I'm missing something, but I don't see why you'd expect a
> different result.  That leaves you with no way to validate the server's
> own certificate.

I don't follow.  Why would the server need to validate it's own
certificate?

--
========================================================================
Ian Pilcher                                         arequipeno@gmail.com
Sometimes there's nothing left to do but crash and burn...or die trying.
========================================================================

On Mar 7, 2013, at 9:37 AM, Ian Pilcher wrote:

> On 03/07/2013 08:28 AM, Tom Lane wrote:
>> Maybe I'm missing something, but I don't see why you'd expect a
>> different result.  That leaves you with no way to validate the server's
>> own certificate.
>
> I don't follow.  Why would the server need to validate it's own
> certificate?

What Tom said works for me.  Here is a page that gives an example and I think it demonstrates that the root CA does not
alloweverybody in the gate, the chain has to be in place: 
 http://stackoverflow.com/questions/1456034/trouble-understanding-ssl-certificate-chain-verification
You can use the "openssl verify" command to test that the root is not wide open on it's own.

On 03/07/2013 12:42 PM, Ray Stell wrote:
> What Tom said works for me.  Here is a page that gives an example and I think it demonstrates that the root CA does
notallow everybody in the gate, the chain has to be in place: 
>  http://stackoverflow.com/questions/1456034/trouble-understanding-ssl-certificate-chain-verification

That page doesn't even mention PostgreSQL.

> You can use the "openssl verify" command to test that the root is not wide open on it's own.

The issue is the behavior of the PostgreSQL server.  "openssl verify" is
germane only in that it points to the source of the problem -- OpenSSL's
insistence on ultimately validating all certificates against a self-
signed root CA.  This requires that the root CA certificate be present
in root.crt, which causes the server to accept connections from all
clients that can present a certificate chain leading to that root CA.

If you don't believe me, test with the attached files, which implement
the following hierarchy.

                    +---------+
                    | Root CA |
                    +---------+
                        /\
                       /  \
                      /    \
                     /      \
                    /        \
                   /          \
                  /            \
                 /              \
          +-----------+    +-----------+
          | Server CA |    | Client CA |
          +-----------+    +-----------+
                /\                \
               /  \                \
              /    \                \
             /      \                \
            /        \                \
           /          \                \
          /            \                \
         /              \                \
  +----------+      +--------+       +--------+
  | postgres |      | "Bad"  |       | "Good" |
  | (server) |      | client |       | client |
  +----------+      +--------+       +--------+

The goal is to configure the server such that the "good" client will
be allowed to connect (because its certificate is signed by the Client
CA), but the "bad" client will not be allowed to connect (because its
certificate is not signed by the Client CA).

You will find the following:

1. You cannot simply use client-ca,crt as $PGDATA/root.crt.  OpenSSL
   will not validate a client certificate without access to the root CA
   certificate.

2. To enable client connections, you must add the root CA certificate
   to $PGDATA/root.crt -- "cat client-ca.crt root-ca.crt > root.crt".

3. Once the root CA certificate is trusted, however, the "bad" client
   can also connect by using a certificate chain that includes the
   Server CA certificate --"cat bad-client.crt server-ca.crt >
   ~/.postgresql/postgresql.crt".

After looking at be-secure.c and investigating the way that OpenSSL
validates certificates, I do not believe that there is any way of
achieving the desired behavior with the current codebase.

Adding pgsql-hackers to see if there is any interest in a patch to add
this functionality.

--
========================================================================
Ian Pilcher                                         arequipeno@gmail.com
Sometimes there's nothing left to do but crash and burn...or die trying.
========================================================================

On 03/09/2013 04:52 PM, Ian Pilcher wrote:
> 3. Once the root CA certificate is trusted, however, the "bad" client
>    can also connect by using a certificate chain that includes the
>    Server CA certificate --"cat bad-client.crt server-ca.crt >
>    ~/.postgresql/postgresql.crt".
>
> After looking at be-secure.c and investigating the way that OpenSSL
> validates certificates, I do not believe that there is any way of
> achieving the desired behavior with the current codebase.
I'm testing this and looking into it now.

At first glance this looks like a genuine problem. We need to be storing
the certs used for validating client cert auth separately from the
certificate chain that links those certs to trusted self-signed CA
roots. I was under the strong impression that OpenSSL would do this if
the client validation certs were in root.crt and the certificate chain
was in OpenSSL's certificate search path and am testing that now. Even
if that's the case we need to at least document this issue and
preferably detect the case where root.crt contains a certificate chain.

If this tests out as expected you need to consider the effects it'd have
on people who're not using self-signed CAs, but are instead using certs
signed by big CAs. *Any other customer of the same CA could potentially
connect to your server with a genuine, valid client cert issued to them
by the CA*. Ouch.

I'm going through and reproducing the problem now and will also test
OpenSSL certificate chain lookup path configurations to see if there's a
way to set things up correctly with the current backend code. I'll
report back shortly.

--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

On 03/09/2013 04:52 PM, Ian Pilcher wrote:
> After looking at be-secure.c and investigating the way that OpenSSL
> validates certificates, I do not believe that there is any way of
> achieving the desired behavior with the current codebase.

Test process:

SET UP SERVER VERIFIED SSL (NO CLIENT CERTS)
------------------------------------------------------------------

Edited postgresql.conf and set:

ssl=on
ssl_cert_file = 'server.crt'
ssl_key_file = 'server.key'
ssl_ca_file='root.crt'

Copied your samples:

# cp $CERTS/postgres.crt server.crt
# cp $CERTS/postgres.key server.key
# cp $CERTS/client-ca.crt root.crt
# chown postgres:postgres root.crt server.crt server.key
# chmod 0600 server.crt server.key root.crt
# systemctl restart  postgresql-9.2.service

$ psql "postgresql://localhost/?sslmode=require"
SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)

(connects OK; expected since we aren't requiring client certs yet and we
aren't validating the server cert).

$ psql "postgresql://localhost/?sslmode=verify-ca"
psql: root certificate file "/home/craig/.postgresql/root.crt" does not
exist
Either provide the file or change sslmode to disable server certificate
verification.

Again expected, though we really should be using the system SSL cert
database. Anyway:

$ mkdir .postgresql
$ cp $CERTS/root-ca.crt ~/.postgresql/root.crt

(This should be the trusted root, not server-ca or client-ca since we
shouldn't have to keep copies of either to verify server trust). Now,
test we can verify the server's identity:

$ psql "postgresql://localhost/?sslmode=require"
psql: SSL error: certificate verify failed

Plonk, we can't. The reason for this is that the server has sent us its
cert and we have the root cert, but we don't have the intermediate
server-ca.crt . Append that to server.crt:

# cat $CERTS/postgres.crt $CERTS/server-ca.crt > server.crt
# systemctl restart  postgresql-9.2.service

$ psql "postgresql://localhost/?sslmode=require"
SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)

OK, we're good now, the server is sending us the intermediate cert we
require. Regular non-client-cert verified SSL is fine.  Examination of
the protocol chat shows that the server is sending a Server Hello with a
Certificate message containing the server and intermdediate certificate DNs:

  id-at-commonName=postgres.example.com,id-at-organizationName=Ian
Pilcher,id-at-localityName=Carrollton,id-at-stateOrProvinceName=Texas,id-at-countryName=US
  id-at-commonName=Server CA,id-at-organizationName=Ian
Pilcher,id-at-localityName=Carrollton,id-at-stateOrProvinceName=Texas,id-at-countryName=US

as expected.


ENABLE CLIENT CERTIFICATE VERIFICATION
---------------------------------------------------------

Now lets install our client certificates in the client.

$ cp $CERTS/good-client.key ~/.postgresql/postgresql.key
$ # Note that the order is important here, the client cert must appear
first, followed by the chain cert(s)
$ cat $CERTS/good-client.crt $CERTS/client-ca.crt >
~/.postgresql/postgresql.crt
$ chmod 0600 .postgresql/postgresql.key

$ psql "postgresql://localhost/?sslmode=verify-ca"
psql: SSL error: tlsv1 alert unknown ca


Examination of the handshake shows that the server is sending a request
for client certificates signed by:

  Distinguished Name: (id-at-commonName=Client
CA,id-at-organizationName=Ian
Pilcher,id-at-localityName=Carrollton,id-at-stateOrProvinceName=Texas,id-at-countryName=US)

and the client is sending in response:

  Certificate (id-at-commonName=Good Client,id-at-organizationName=Ian
Pilcher,id-at-localityName=Carrollton,id-at-stateOrProvinceName=Texas,id-at-countryName=US)
  Certificate (id-at-commonName=Client CA,id-at-organizationName=Ian
Pilcher,id-at-localityName=Carrollton,id-at-stateOrProvinceName=Texas,id-at-countryName=US)

as expected, and good-client.crt is indeed signed by client-ca.crt .
Again the issue appears to be that Pg can't find the root of trust,
which is fair enough given that it is not present in Pg's root.crt or
server.crt and it isn't installed system-wide either. We could add it to
the server's root.crt but that'd cause the issues that started this thread:

# cat $CERTS/client-ca.crt $CERTS/root-ca.crt > root.crt
# systemctl restart  postgresql-9.2.service

the good client cert works now:

$ psql "postgresql://localhost/?sslmode=verify-ca"
SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)

PROBLEM VERIFIED
--------------------------

... but so does the one we don't want to trust, as per the problem report:

$ cat $CERTS/bad-client.crt $CERTS/server-ca.crt > postgresql.crt
$ cp $CERTS/bad-client.key postgresql.key
$ chmod 600 postgresql.key
$ psql "postgresql://localhost/?sslmode=verify-ca"
SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)

So this problem is verified.


THE SOLUTION
--------------------

What we need to happen instead is for root.crt to contain only the
trusted certificates and have a *separate* file or directory for
intermediate certificates that OpenSSL can look up to get the
intermediates it needs to validate client certs, like
`ssl_ca_chain_file` or `ssl_ca_chain_path` if we want to support
OpenSSL's hashed certificate directories.

System wide installation of the root may allow OpenSSL to discover it
and use it for verification back to the root without having to trust it
to sign clients. I'll do some more checking to see if this is possible
with how Pg uses OpenSSL but I'm inclined to doubt it.

I thought you might be able to add the common root to the server.crt
certificate chain to let OpenSSL discover it that way, but it looks like
OpenSSL won't use certs it's seen in server.crt when verifying client
cert trust paths.

--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

On 03/18/2013 12:07 AM, Craig Ringer wrote:
> So this problem is verified.

Thanks for taking the time to look into this.  Good to know I'm not
crazy.

> What we need to happen instead is for root.crt to contain only the
> trusted certificates and have a *separate* file or directory for
> intermediate certificates that OpenSSL can look up to get the
> intermediates it needs to validate client certs, like
> `ssl_ca_chain_file` or `ssl_ca_chain_path` if we want to support
> OpenSSL's hashed certificate directories.

I did a fair bit of searching and asked about this subject on the
openssl-users list.  The consensus seems to be that doing this with
OpenSSL will require 2 separate sets of certificates and a validation
callback.

The two sets of certificates are:

* Trusted certificates - What currently goes in the (unfortunately
  named) root.crt file.

* Validation-only certificates - CA certificates that are used only to
  complete the chain from a trusted certificate to a self-signed root.
  I haven't been able to come up with a particularly good name for a
  file containing this type of certificate(s) -- validate.crt?

All of the certificates in both sets get added to the SSL_CTX, so that
OpenSSL can do its normal validation -- all the way to a root CA.  The
trusted certificates also need to be maintained as a separate set
(X509_STORE?).

Once OpenSSL has built the complete certificate chain, the validation
callback can refuse the connection if the chain does not contain at
least one of the *trusted* certificates.

This is conceptually simple, and I've been fiddling with it for the last
week or so.  Unfortunately, the OpenSSL documentation has made this far
more challenging that it should be.  Simple things like reading multiple
certificates from a file, checking whether an X509_STORE contains a
particular certificate, etc. are all proving to be unexpectedly
difficult.  (I never thought that I'd miss the Java SSL API!)

> System wide installation of the root may allow OpenSSL to discover it
> and use it for verification back to the root without having to trust it
> to sign clients. I'll do some more checking to see if this is possible
> with how Pg uses OpenSSL but I'm inclined to doubt it.

Me too.  I think that OpenSSL's behavior embodies the idea that a
certificate can only be validated if a chain can be formed to a self-
signed root CA.  (And there's probably a pretty good argument to be
made for this position, particularly when CRLs are added to the mix.)

> I thought you might be able to add the common root to the server.crt
> certificate chain to let OpenSSL discover it that way, but it looks like
> OpenSSL won't use certs it's seen in server.crt when verifying client
> cert trust paths.

Nope.  It's pretty obvious from be-secure.c that only the certificates
in root.crt will be used.

--
========================================================================
Ian Pilcher                                         arequipeno@gmail.com
Sometimes there's nothing left to do but crash and burn...or die trying.
========================================================================

On 03/18/2013 01:07 PM, Craig Ringer wrote:
> System wide installation of the root may allow OpenSSL to discover it
> and use it for verification back to the root without having to trust it
> to sign clients. I'll do some more checking to see if this is possible
> with how Pg uses OpenSSL but I'm inclined to doubt it.
It looks like we aren't reading the system-wide certs or looking them up
when certs aren't resolved in the files Pg explicitly passes to OpenSSL,
so a system-wide install doesn't look like it'll work.

I've had a look at how Apache handles this and it took me a while to
work out what's going on. Apache uses SSLCACertificateFile (concatenated
certs) / SSLCACertificatePath (hashed dir) to look up trusted
certificate signers. It According to the docs it doesn't appear to make
any provision there for trusting intermediate certificates but not their
parents as signers of client certificates, but it looks like support is
there, the docs just don't explain it well.

Apache has SSLCADNRequestFile / SSLCADNRequestPath which are described
as controlling the acceptable certificate names sent to clients in a
client cert request. The docs don't make it clear if Apache will trust
only client certs with these certs in their chains or whether this only
controls the list of certificate DNs presented to the client rather than
what's accepted in response. The code suggests that they control trust
not just the cert list presented.

In Apache's modules/ssl/ssl_engine_init.c it calls
SSL_CTX_load_verify_locations on the SSLCACertificateFile and
SSLCACertificatePath.

It then  calls ssl_init_FindCAList with the
SSLCADNRequestFile/SSLCADNRequestPath if they're specified in the
configuration, otherwise it calls it with the
SSLCACertificateFile/SSLCACertificatePath . That calls
ssl_init_PushCAList on all of the certs it finds in the File and Path
variants. For each cert file that calls SSL_load_client_CA_file and for
each cert within each file pushes the cert onto a STACK_OF(X509_NAME) if
a cert with the same DN isn't already in the stack. It passes the stack
to OpenSSL's SSL_CTX_set_client_CA_list .

So what Apache does appears to boil down to:

SSL_CTX_load_verify_locations(ca_file,ca_path);
if (ca_dn_file || ca_dn_path) {
    SSL_CTX_set_client_CA_list( ... STACK_OF unique certs on ca_dn_file
and ca_dn_path ... );
} else {
    SSL_CTX_set_client_CA_list( ... STACK_OF unique certs on ca_file and
ca_path );
}

This appears to match Ian's description of having a validation-only cert
list and a separate list of certs used to verify clients. I'd like to
follow Apache's model:

in postgresql.conf, if ssl_ca_file is set then pass it to
SSL_CTX_load_verify_locations . If the proposed new parameter
ssl_ca_valid_client_signers_file is set then pass the certs in that to
SSL_CTX_set_client_CA_list ; otherwise pass the certs in ssl_ca_file to
SSL_CTX_set_client_CA_list and thus retain the current behaviour.
Hopefully we can avoid the ugly read-and-deduplicate stuff Apache has to
do because we currently only support a certfile anyway, we don't read
certdirs, so I'll look for helper functions that wrap
SSL_CTX_set_client_CA_list.

--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

On 03/18/2013 02:27 PM, Ian Pilcher wrote:
> On 03/18/2013 12:07 AM, Craig Ringer wrote:
>> So this problem is verified.

> * Trusted certificates - What currently goes in the (unfortunately
>   named) root.crt file.

Well, a little unfortunate. It contains roots of *client authentication*
trust, which is fair enough, they just aren't necessarily self-signed
certificates that are roots of *certificate validity* trust (root CA certs).

This list is set by SSL_CTX_set_client_CA_list . The examples section of
its man page contains:

  Scan all certificates in CAfile and list them as acceptable CAs:

       SSL_CTX_set_client_CA_list(ctx,SSL_load_client_CA_file(CAfile));

> * Validation-only certificates - CA certificates that are used only to
>   complete the chain from a trusted certificate to a self-signed root.
>   I haven't been able to come up with a particularly good name for a
>   file containing this type of certificate(s) -- validate.crt?

We should probably take advantage of the fact that 9.2 made these
filenames configurable to deprecate root.crt and choose two descriptive
filenames, something like trusted_cert_roots.crt and
trusted_client_cert_signers.crt .

> This is conceptually simple, and I've been fiddling with it for the last
> week or so.  Unfortunately, the OpenSSL documentation has made this far
> more challenging that it should be.  Simple things like reading multiple
> certificates from a file, checking whether an X509_STORE contains a
> particular certificate, etc. are all proving to be unexpectedly
> difficult.  (I never thought that I'd miss the Java SSL API!)

Apache's sources are useful there. When working with OpenSSL sometimes
the sanest option is to find something you know already does it right,
work out how, *understand why it works* and then apply that approach to
your code. Blindly copying their approach is stupid and guaranteed to
lead to security holes, but others' code remains some of the best
documentation for OpenSSL if used for hints rather than blindly copied.

--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Craig, all,

* Craig Ringer (craig@2ndquadrant.com) wrote:
> PROBLEM VERIFIED

Let me just say "ugh".  I've long wondered why we have things set up in
such a way that the whole chain has to be in one file, but it didn't
occur to me that it'd actually end up causing this issue.  In some ways,
I really wonder about this being OpenSSL's fault as much as ours, but I
doubt they'd see it that way. :)

> What we need to happen instead is for root.crt to contain only the
> trusted certificates and have a *separate* file or directory for
> intermediate certificates that OpenSSL can look up to get the
> intermediates it needs to validate client certs, like
> `ssl_ca_chain_file` or `ssl_ca_chain_path` if we want to support
> OpenSSL's hashed certificate directories.

Makes sense to me.  I'm not particular about the names, but isn't this
set of CAs generally considered intermediary?  Eg: 'trusted', '
intermediate', etc?

    Thanks,

        Stephen

On 03/18/2013 02:01 AM, Craig Ringer wrote:
> This appears to match Ian's description of having a validation-only cert
> list and a separate list of certs used to verify clients. I'd like to
> follow Apache's model:

Ready for some more good news?

It's possible that I'm missing something, but Apache (mod_ssl) appears
to exhibit the exact same behavior.

Tested on Fedora 18, with the following packages:

  httpd-2.4.3-15.fc18.x86_64
  mod_ssl-2.4.3-15.fc18.x86_64
  openssl-1.0.1e-3.fc18.x86_64

I have set the following in /etc/httpd/conf.d/ssl.conf:

  Listen 44443 https
  <VirtualHost _default_:44443>
  ServerName postgres.example.com
  SSLCertificateFile /etc/pki/tls/certs/postgres.crt
  SSLCertificateKeyFile /etc/pki/tls/private/postgres.key
  SSLCACertificateFile /etc/pki/tls/certs/client-ca.chain
  SSLCADNRequestFile /etc/pki/tls/certs/client-ca.crt
  SSLVerifyClient require
  SSLVerifyDepth  10

Notes:

  * The port is set to 44443, because that's hard-coded into the
    (attached) test client.
  * I am using the certificates that I previously sent.
  * ServerName is set to postgres.example.com to match its certificate.
  * postgres.crt contains its entire chain (postgres.crt + server-ca.crt
    + root-ca.crt), so that I don't have to specify a
    SSLCertificateChainFile.
  * client-ca.chain is client-ca.crt + root-ca.crt.  As with PostgreSQL,
    I found that I have to provide the root CA certificate in order for
    any client to connect.

With this configuration, the test client is able to connect with the
"good" client certificate, but it is also able to connect with the "bad"
client certificate when it presents a certificate chain that includes
the server CA certificate.

--
========================================================================
Ian Pilcher                                         arequipeno@gmail.com
Sometimes there's nothing left to do but crash and burn...or die trying.
========================================================================

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/18/2013 08:55 PM, Stephen Frost wrote:
> Makes sense to me. I'm not particular about the names, but isn't this
> set of CAs generally considered intermediary? Eg: 'trusted', '
> intermediate', etc?
They are intermediary, but we're dealing with the case where trust and
authorization are not the same thing. Trust stems from the trusted root
in the SSL CA model, but that's a chain of trust for *identity*
(authentication), not *authorization*.

Bob J. Criminal might well have a client certificate from a trusted
authority proving that he's who he says he is (he's authenticated) but
we sure as hell don't want to authorize his access to anything.

That's where the intermediate certs come in. We might say "Only users
with certificates issued by our corporate HR team are authorized to
connect to our servers". This is a root of trust, but this time it's a
root of trust to *authorize*, not just to authenticate.

The usual SSL terminology doesn't consider this, because it's a simple
back and white trust model where authenticated = authorized.

I guess that suggests we should be calling this something like
'ssl_authorized_client_roots'.

- --
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRR/dqAAoJELBXNkqjr+S2TV4H/3f9Hnf9JhSuGhWblh2adgTJ
Rkdx/9RbByJDMJP0s0c8C1sXaWZGJmKmLhJoes4IIvOVW85SVUa9WoT+UBJPdx9P
esUNsSLFokLqom3TxNRZOHaloyZ+OZafSUnKCwMOIvD0hIehrS3Wcg70QMSj06tX
h22BVhA8bzO1Wdg9UdD98jcuWdEbLgWzVtvIXjICcMJ1azgiF1VY4zwUUbBJBfLG
UIA7+2TtVaXQuge6qWgId0RTKKrb6cLHXCSQ/rigy0mRH9m/G5jKmqENvLAnafI4
4lSBPyDzNj2fBfP9YgIiAe/EGjnJMWQfBBghQI3QrK2kjOZXtzZoOb4XEjfn3FI=
=u+2j
-----END PGP SIGNATURE-----

Craig,

* Craig Ringer (craig@2ndquadrant.com) wrote:
> They are intermediary, but we're dealing with the case where trust and
> authorization are not the same thing. Trust stems from the trusted root
> in the SSL CA model, but that's a chain of trust for *identity*
> (authentication), not *authorization*.

Oh, I see.

> The usual SSL terminology doesn't consider this, because it's a simple
> back and white trust model where authenticated = authorized.

That's not entirely accurate on a couple of levels.  First, basic SSL is
only concerned with authentication, authorization has historically been
left up to the application.  The more typical approach with the
situation you're describing is to have an organization-level root CA
which you only issue certs against.  If you're using a public CA as your
root, then you need to make sure you know how to ensure only the right
people have access, typically be storing in the mapping table the unique
ID issued by the CA for your users.  It's very rare, from what I've
seen, for public CAs to issue intermediate CAs to organizations to
generate their own certs off of, so I'm a bit confused about how we got
to this point.

What I *have* seen is cross-root-cert trusts (known as the Federal
Bridge in the US government), but that's quite a different thing as you
have multiple self-signed root CAs involved and need to know how to
properly traverse between them based on the trusts which have been
built.

Regarding cross-CAS authorization, there are extended attributes which
are listed in the certificates that applications are expected to look
at when considering authorization for the client.  It's been taken
further than that however, where inter-CA trusts have been defined
with actual mappings between these extended attributes, including
'null' mappings (indicating that CA 'A' doesn't trust attribute 'q'
from CA 'B').

> I guess that suggests we should be calling this something like
> 'ssl_authorized_client_roots'.

I'm no longer convinced that this really makes sense and I'm a bit
worried about the simple authentication issue which I thought was at the
heart of this concern.  Is there anything there that you see as being an
issue with what we're doing currently..?

I do think we want to figure out a way to improve our mapping table to
be able to use more than just the CN, since that can be repeated in
multiple certs issued from a root CA, particularly when there are
intermediary CAs.  One option might be to provide a way to map against a
specific issuing CA, or to a CA in the chain, but there's a lot of risk
to that due to CA churn (in large setups, you're going to have lots of
users who have certs issued from a bunch of different CAs, and those
user certs will roll to new CAs as new badges are issued, for
example..).  It can get to be a real nightmare to try and keep up with
all of the changes at that level.

    Thanks,

        Stephen

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/19/2013 01:46 PM, Stephen Frost wrote:
> If you're using a public CA as your
> root, then you need to make sure you know how to ensure only the right
> people have access, typically be storing in the mapping table the unique
> ID issued by the CA for your users.
Yep, in most applications I've seen you usually store a list of
authorized SubjectDNs or you just use your own self-signed root and
issue certs from it.

I'm pretty sure I've seen tools match on part of the DN, like the
organisation field, but since I can't remember *where* I'm not sure
that's all that useful.
> It's very rare, from what I've
> seen, for public CAs to issue intermediate CAs to organizations to
> generate their own certs off of, so I'm a bit confused about how we got
> to this point.
I don't know about "very rare" but it's certainly not common outside
very large orgs. I tried to find a CA that'd let me issue intermediate
client certs for 2ndQuadrant but found nobody that'd do it for
certificate volumes less than several thousand new certs a month. I'd
been using intermediate CAs based on my own self-signed CA root quite
heavily in infrastructure elsewhere I was rather surprised that the same
sort of thing wasn't easily available for public CAs.

I get the impression it's fairly common in internal infrastructure,
especially with the cert management tools offered by Microsoft Active
Directory servers, but have no strong information to substantiate this.
Nor do I know whether we need to support this mode of operation.

BTW, This discussion has made me realise that I know less about SSL/TLS
and X.509 certificate extensions than I'd like to when dealing with this
topic. In particular, I don't know whether a CA can issue an
intermediate CA with extensions that restrict it to validly signing only
host certificates for hosts under a particular domain or
user-identifying client certs with CNs under a particular organisation -
and whether, if such extensions exist, applications actually check them
when verifying the certificate trust chain.


> What I *have* seen is cross-root-cert trusts (known as the Federal
> Bridge in the US government), but that's quite a different thing as you
> have multiple self-signed root CAs involved and need to know how to
> properly traverse between them based on the trusts which have been
> built.
Ugh, that's not something I've ever had the ... privilege ... to deal
with before.
>
> I'm no longer convinced that this really makes sense and I'm a bit
> worried about the simple authentication issue which I thought was at the
> heart of this concern. Is there anything there that you see as being an
> issue with what we're doing currently..?
Only for using intermediate certs as authorization roots, and it may be
reasonable to say "we don't support that, use an authorized DN list". Or
come up with a better solution like checking attributes of the SubjectDN
for authorization purposes after validating the signature chain to prove
authenticity.
> I do think we want to figure out a way to improve our mapping table to
> be able to use more than just the CN, since that can be repeated in
> multiple certs issued from a root CA, particularly when there are
> intermediary CAs. One option might be to provide a way to map against a
> specific issuing CA, or to a CA in the chain, but there's a lot of risk
> to that due to CA churn (in large setups, you're going to have lots of
> users who have certs issued from a bunch of different CAs, and those
> user certs will roll to new CAs as new badges are issued, for
> example..). It can get to be a real nightmare to try and keep up with
> all of the changes at that level.
Certificate fingerprint? Easily obtained via most client UIs and via
openssl x509 -in cert.crt -fingerprint, eg:

SHA1 Fingerprint=DA:03:9B:FB:81:69:AB:48:64:3D:35:B4:90:56:CF:F1:24:FE:89:B0

However, if I was managing a group large enough to want cert auth I'd
want to be able to specify something like:

    SubjectDNMatches: C=*, ST=*, L=*, O=MyCompany, CN=*

... in which case there'd no longer be a need to restrict trust to
intermediate CAs, you'd just trust the root and restrict the authorized
SubjectDNs. If you don't trust your own root CA not to issue certs in
your company's name to 3rd parties you shouldn't be using it. (Whether
it's actually sane to trust a CA is another argument).

- --
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRSAJZAAoJELBXNkqjr+S2JrcIALZebfEW4FbfFI6WOs6qDutr
tz486SlnPV+cf29ex242evSUgNQTz38uFKMIs9EIRfe7sVKz3whn0MARmQY9dKph
CusbXNqcPIBbZIZM1hObaKOnMvNnGk5sxnRh4iKjzcMjqCULG5LVX7bXAXn3PcjA
u3lYlNWONWdmz708QOCgvpui4wEv5+bVuik/CnRdPu+BWAcndJHUMuxZMxkUC/rs
4OjLlEg6BPiXRgIKTFBNsa0vvCyVBUd5ri0RCtxUr5T/L/ORWdM+Ic0nqCEPTqyI
EOtDKuNZUqEnsCOacwulwRDxQXnUoU+6zBHud8al32+PeWLKHLAIcFEiYYSQUjI=
=Ba3M
-----END PGP SIGNATURE-----

On Tue, Mar 19, 2013 at 01:46:32AM -0400, Stephen Frost wrote:
> > I guess that suggests we should be calling this something like
> > 'ssl_authorized_client_roots'.
>
> I'm no longer convinced that this really makes sense and I'm a bit
> worried about the simple authentication issue which I thought was at the
> heart of this concern.  Is there anything there that you see as being an
> issue with what we're doing currently..?

I too am worried that make SSL even more flexible will make simple setups
more complex to setup.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Craig,

* Craig Ringer (craig@2ndquadrant.com) wrote:
> Yep, in most applications I've seen you usually store a list of
> authorized SubjectDNs or you just use your own self-signed root and
> issue certs from it.

Even with a self-signed root issuing certs, you need to map the
individual cert to a PG user in some fashion.

> I'm pretty sure I've seen tools match on part of the DN, like the
> organisation field, but since I can't remember *where* I'm not sure
> that's all that useful.

Looking at what other tools do and how they handle this question would
certainly be a good idea.

> I don't know about "very rare" but it's certainly not common outside
> very large orgs. I tried to find a CA that'd let me issue intermediate
> client certs for 2ndQuadrant but found nobody that'd do it for
> certificate volumes less than several thousand new certs a month. I'd
> been using intermediate CAs based on my own self-signed CA root quite
> heavily in infrastructure elsewhere I was rather surprised that the same
> sort of thing wasn't easily available for public CAs.

It's pretty simple, really- issuing certs is how the public CAs make
their money.  If they give you a CA cert that can issue certs, they're
cut out of the loop.

> I get the impression it's fairly common in internal infrastructure,
> especially with the cert management tools offered by Microsoft Active
> Directory servers, but have no strong information to substantiate this.
> Nor do I know whether we need to support this mode of operation.

In general, CAs view intermediate certs as a way to provide automated
systems while having their self-signed root private key highly
protected.  The intermediate CAs therefore have a shorter life-span and
end up changing much more frequently than the root certs (though root
certs certainly also do change, but it's much more painful).  That's one
of the reasons that they're bad to use as part of the authentication
criteria.

The problem with trying to use intermediate CAs as a way of dividing up
organizational responsibility is simply that there's very few systems
out there which will support that kind of configuration, from what I've
seen.  Wrt Active Directory this problem is actually very well solved
through use of Kerberos, where you have multiple realms, directional
trust between the realms, and most tools (including PG) understand that
a principal is the combination of username@REALM and let you authorize
based on that.

> BTW, This discussion has made me realise that I know less about SSL/TLS
> and X.509 certificate extensions than I'd like to when dealing with this
> topic. In particular, I don't know whether a CA can issue an
> intermediate CA with extensions that restrict it to validly signing only
> host certificates for hosts under a particular domain or
> user-identifying client certs with CNs under a particular organisation -
> and whether, if such extensions exist, applications actually check them
> when verifying the certificate trust chain.

You can set flags on certificates but I've not seen the kind of complex
restrictions that you're describing.  Remember that anything that the CA
sets for an intermediate CA cert would have to be checked by whatever
software is doing the certificate validation, meaning that you'd have to
make sure all your certificate-based software is updated to do that kind
of validation (and do it in a consistent way, or you'd get all kinds of
fun failures).

> Ugh, that's not something I've ever had the ... privilege ... to deal
> with before.

I worked for the company that built the original federal bridge
software. :)  I wasn't directly involved in that project, but I
certainly gained some understanding of the complexities through working
with the folks who were.

> Only for using intermediate certs as authorization roots, and it may be
> reasonable to say "we don't support that, use an authorized DN list". Or
> come up with a better solution like checking attributes of the SubjectDN
> for authorization purposes after validating the signature chain to prove
> authenticity.

I think it'd be valuable to distinguish "trusted CAs" from "intermediate
CAs" in PG explicitly (as I recall, you can already do this by simply
ensuring that your OpenSSL config is set up correctly for the system
wide defaults).  That's what most serious SSL users will be familiar
with anyway.  I'm on the fence about if only supporting a list of
"trusted CAs" (through the cert files that we currently have) rises to
the level of being a security issue, but we should at least update the
documentation to reflect that all CAs listed in the file are fully
trusted and plan to provide an intermediate CA list option.  To be
honest, our entire SSL support mechanism could really use some work and
it'd be great if we had some folks looking into it seriously.  One of
the problems we've long had is the dependency on OpenSSL (yes, it's a
problem) and it'd be good to consider, if we're changing the SSL
configuration, how we could make it easier to introduce other SSL
libraries in the future.

> Certificate fingerprint? Easily obtained via most client UIs and via

They change too much.  This is the trade-off that you end up having to
deal with- either you depend on the DN/CN, which can end up being
duplicately issued by a given root CA, or you depend on the fingerprint
and then have to re-enroll individuals whenever they get a new cert.
This isn't an easy problem to solve, when dealing with the US ACES
program, we could never get them to give us a "unique-to-a-person"
identifier, the best we got was the DN.  That changes when an individual
changes organizations, which happens stupidly frequently in government,
but (on rare occation..) it may also come with an actual change in
responsibility, such that the individual shouldn't have access any
longer.  Being able to map certs based on the DN instead of just the CN
would, imv, be a valuable improvment to the PG SSL options.  We'd
probably want to support that on a per-map basis and not just globally.

> However, if I was managing a group large enough to want cert auth I'd
> want to be able to specify something like:
>
>     SubjectDNMatches: C=*, ST=*, L=*, O=MyCompany, CN=*

If we had a way to use the DN in pg_ident.conf, you could use a regex to
do the matching.

> ... in which case there'd no longer be a need to restrict trust to
> intermediate CAs, you'd just trust the root and restrict the authorized
> SubjectDNs.

In general, I'm open to doing it all kinds of ways; more options here
would be better than what we've currently got.  I would put splitting
the CA cert chain file into trusted-CAs and intermediate-CAs as the
higher priority here, followed by allowing the use of the DN for the
mapping to PG user, and perhaps lastly something that tried to tie
into a specific path, chain, or intermediate CA, as I don't see a very
big use case for that.

    Thanks,

        Stephen

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/19/2013 08:39 PM, Stephen Frost wrote:
> Craig,
>
> * Craig Ringer (craig@2ndquadrant.com) wrote:
>> Yep, in most applications I've seen you usually store a list of
>> authorized SubjectDNs or you just use your own self-signed root and
>> issue certs from it.
>
> Even with a self-signed root issuing certs, you need to map the
> individual cert to a PG user in some fashion.
>

The more I look a this, the more it looks like trying to use
intermediate CAs as authentication roots is largely wrong anyway. We
should document this with something like:

NOTE: Only self-signed root CA certificates should be added to
ssl_ca_file. If you add an intermediate CA certificate (one that's not
self-signed) then PostgreSQL will not be able to validate client
certificates against it because it will not have access to the full
certificate chain. You can't fix that by adding the full certificate
chain then PostgreSQL will then accept client certificates trusted by
any member of the chain, including the root, so the effect is the same
as placing only the root certificate in the file. It is not currently
possible to trust certificates signed by an intermediate CA but not the
parents in its certificate chain.

... plus some explanation that having a valid trusted cert doesn't mean
you're authorized for access, you still have to meet the requrements in
pg_hba.conf, have a valid username/password or match an authorized
certificate DN (depending on config), etc.

As far as I'm concerned that's the immediate problem fixed. It may be
worth adding a warning on startup if we find non-self-signed certs in
root.crt too, something like 'WARNING: Intermediate certificate found in
root.crt. This does not do what you expect and your configuration may be
insecure; see the Client Certificates chapter in the documentation.'



We could then look at using more flexible approaches to match PostgreSQL
users to client certificates, like regexps or (preferably, IMO)
DN-component based solutions to extract usernames from cert DNs, etc.
Various ways to specify *authorization*.

It's looking more and more like the *authentication* side is basically
"do you trust this CA root not to sign certs for fraudlent/fake
SubjectDNs or issue intermediate certs that might do so? Trust: include
it. No trust: Don't." That's what we have now, it just needs to be
explained better in the docs.

- --
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRSGoOAAoJELBXNkqjr+S26esIALSmgX6/4lC+J7W3YPDpl1DE
UJsSGc46oBZbC/5xDwBELh2Tg+fqzIe+Kmx+EpsC20MaGinqEz9iwTb2M7vTFhxh
nvAkp1Em8MhR6lvCKITjPnDBCv7yQ7K3yTAfHO+LU2J1t3eVhStpXh71/73pRLoQ
p3SAUwO0EBnZFdY2HVLPABK7tpjuf5Mpn0QFR9T+KvsgcP9QXiV0UTFI0IxlQrpE
NRlJfPwkoYAweISTACrDwqJHJ3sL/qLdOQ8l4BCsiwtqynX7fPhxmDUuBXrOTqlS
dwW9ZkBJ9jvXjF3PPk1t0oujlMJGBC4Y7xgIb0Kd87Vyv/OTkWE4XKriDhIH6oQ=
=f3qr
-----END PGP SIGNATURE-----

* Craig Ringer (craig@2ndquadrant.com) wrote:
> As far as I'm concerned that's the immediate problem fixed. It may be
> worth adding a warning on startup if we find non-self-signed certs in
> root.crt too, something like 'WARNING: Intermediate certificate found in
> root.crt. This does not do what you expect and your configuration may be
> insecure; see the Client Certificates chapter in the documentation.'

I'm not sure that I follow this logic, unless you're proposing that
intermediate CAs only be allowed to be picked up from system-wide
configuration?  That strikes me as overly constrained as I imagine there
are valid configurations today which have intermediate CAs listed, with
the intention that they be available for PG to build the chain from a
client cert that is presented back up to the root.  Now, the client
might be able to provide such an intermediate CA cert too (one of the
fun things about SSL is that the client can send any 'missing' certs to
the server, if it has them available..), but it also might not.

> We could then look at using more flexible approaches to match PostgreSQL
> users to client certificates, like regexps or (preferably, IMO)
> DN-component based solutions to extract usernames from cert DNs, etc.
> Various ways to specify *authorization*.

Sure.

> It's looking more and more like the *authentication* side is basically
> "do you trust this CA root not to sign certs for fraudlent/fake
> SubjectDNs or issue intermediate certs that might do so? Trust: include
> it. No trust: Don't." That's what we have now, it just needs to be
> explained better in the docs.

I certainly agree that the docs could be improved in this area. :)

    Thanks,

        Stephen

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/19/2013 09:46 PM, Stephen Frost wrote:
> * Craig Ringer (craig@2ndquadrant.com) wrote:
>> As far as I'm concerned that's the immediate problem fixed. It may be
>> worth adding a warning on startup if we find non-self-signed certs in
>> root.crt too, something like 'WARNING: Intermediate certificate found in
>> root.crt. This does not do what you expect and your configuration may be
>> insecure; see the Client Certificates chapter in the documentation.'
>
> I'm not sure that I follow this logic, unless you're proposing that
> intermediate CAs only be allowed to be picked up from system-wide
> configuration? That strikes me as overly constrained as I imagine there
> are valid configurations today which have intermediate CAs listed, with
> the intention that they be available for PG to build the chain from a
> client cert that is presented back up to the root. Now, the client
> might be able to provide such an intermediate CA cert too (one of the
> fun things about SSL is that the client can send any 'missing' certs to
> the server, if it has them available..), but it also might not.
>

Drat, you're quite right. I've always included the full certificate
chain in client certs but it's in no way required.

I guess that pretty much means mainaining the status quo and documenting
it better.

- --
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRSp3fAAoJELBXNkqjr+S2+JYH+wUo2mCMB2n3/mXo24l0rO5+
mxS6d9uJNIZZErZX2I/NfY59kLX1ypUAeGhQnCSOZuxig6Xd91nXzRdkaQF/+WHa
9hEAXbOtl7bMgj8cEIfloQlSU94VXamH53i5YL5ZVLqkQG/7uknY05NbJs3IGM5g
ALrEgo3XOC8JyUz21hZzaQOb2vbdSh0F0O17EoJz1fLY6l5ScFnLWihKYurp5Oq0
em1bsN0GKckmSa7a9mJ37Hvowi92epbtF4XR1DyrQGOHQSCLq0NnCthA5MtdPXN0
+BJQWZfx0qcRcrHMILkFa0Uu7Bc9Ao0q06l55DNSyYXx1FWN0cBArGpXcoPb8Zs=
=BAYd
-----END PGP SIGNATURE-----

On Thu, Mar 21, 2013 at 01:42:55PM +0800, Craig Ringer wrote:
> On 03/19/2013 09:46 PM, Stephen Frost wrote:
> > * Craig Ringer (craig@2ndquadrant.com) wrote:
> >> As far as I'm concerned that's the immediate problem fixed. It may be
> >> worth adding a warning on startup if we find non-self-signed certs in
> >> root.crt too, something like 'WARNING: Intermediate certificate found in
> >> root.crt. This does not do what you expect and your configuration may be
> >> insecure; see the Client Certificates chapter in the documentation.'
> >
> > I'm not sure that I follow this logic, unless you're proposing that
> > intermediate CAs only be allowed to be picked up from system-wide
> > configuration? That strikes me as overly constrained as I imagine there
> > are valid configurations today which have intermediate CAs listed, with
> > the intention that they be available for PG to build the chain from a
> > client cert that is presented back up to the root. Now, the client
> > might be able to provide such an intermediate CA cert too (one of the
> > fun things about SSL is that the client can send any 'missing' certs to
> > the server, if it has them available..), but it also might not.
> >
>
> Drat, you're quite right. I've always included the full certificate
> chain in client certs but it's in no way required.
>
> I guess that pretty much means mainaining the status quo and documenting
> it better.

I have developed the attached patch to document this behavior.  My goals
were:

* clarify that a cert can match a remote intermediate or root certificate
* clarify that the client cert must match a server root.crt
* clarify that the server cert much match a client root.crt
* clarify that the root certificate does not have to be specified
  in the client or server cert as long as the remote end has the chain
  to the root

Does it meet these goals?  Is it correct?

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + Everyone has their own god. +