Thread: Correction of intermediate certificate handling
We have been confused by the behavior of intermediate certificates in
Postgres for many years. Some people put the intermediate certificates
only on the server and they were supplied to the client, while other
people couldn't get that to work. In our documentation we recommended
storing intermediate certificates on the client and server.
As part of research for my security talks:
https://momjian.us/main/presentations/security.html
I asked Stephen Frost and David Steele for details on the arcane art of
SSL certificate creation. They showed me scripts they use and explained
that they properly pass intermediate certificates to clients. The trick
was to use the v3_ca extension when creating root and intermediate
certificates.
My talk documents this behavior. In this talk:
https://momjian.us/main/writings/pgsql/tls.pdf
slide 47 and 49 use -extensions v3_ca. Slides 73 and 74 show that the
intermediate is not needed on the client if it is created with v3_ca and
exist on the server. Slide 75 shows that the server certificate must be
first in server.crt.
I have created the attached doc patch to add this information to our
docs. I would like to backpatch this since what we have now, while it
works, is inaccurate.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +
Attachment
On Mon, Jan 15, 2018 at 07:22:38PM -0500, Bruce Momjian wrote: > I asked Stephen Frost and David Steele for details on the arcane art of > SSL certificate creation. They showed me scripts they use and explained > that they properly pass intermediate certificates to clients. The trick > was to use the v3_ca extension when creating root and intermediate > certificates. > > My talk documents this behavior. In this talk: > > https://momjian.us/main/writings/pgsql/tls.pdf > > slide 47 and 49 use -extensions v3_ca. Slides 73 and 74 show that the > intermediate is not needed on the client if it is created with v3_ca and > exist on the server. Slide 75 shows that the server certificate must be > first in server.crt. > > I have created the attached doc patch to add this information to our > docs. I would like to backpatch this since what we have now, while it > works, is inaccurate. I have spent some time looking at your patch, this gets a +1 from here. This bit is important. I am happy that your patch mentions that intermediate certificates avoid the need to store root ones on the client. Should the docs mention terms like "chain of trust"? Perhaps the docs could also include an example of command to create a root and an intermediate certificate in runtime.sgml or such? On top of that, src/test/ssl does not provide any kind of coverage for that. It would be an area of improvement for those tests. -- Michael
Attachment
On Tue, Jan 16, 2018 at 02:33:05PM +0900, Michael Paquier wrote: > > My talk documents this behavior. In this talk: > > > > https://momjian.us/main/writings/pgsql/tls.pdf > > > > slide 47 and 49 use -extensions v3_ca. Slides 73 and 74 show that the > > intermediate is not needed on the client if it is created with v3_ca and > > exist on the server. Slide 75 shows that the server certificate must be > > first in server.crt. > > > > I have created the attached doc patch to add this information to our > > docs. I would like to backpatch this since what we have now, while it > > works, is inaccurate. > > I have spent some time looking at your patch, this gets a +1 from here. Thanks. > This bit is important. I am happy that your patch mentions that > intermediate certificates avoid the need to store root ones on the > client. Should the docs mention terms like "chain of trust"? I think the question is how much do we want to "teach" people in our docs. We do oddly but wisely link from our docs to HP OpenVMS docs about how the chain of trust works: http://h41379.www4.hpe.com/doc/83final/ba554_90007/ch04s02.html I will write up a paragraph about the concepts for our docs for the group's review. > Perhaps the docs could also include an example of command to create a > root and an intermediate certificate in runtime.sgml or such? Yes, I have thought about that. My presentation has clear examples that we can use, again based on Stephen and David's scripts using v3_ca. I will work up a possible patch for that too. > On top of that, src/test/ssl does not provide any kind of coverage for > that. It would be an area of improvement for those tests. Wow, I have no idea how to do that. Let me look. Seems I have more work to do. Instead of appending to this doc patch, I will work on a second one for review. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
On Tue, Jan 16, 2018 at 11:21:22AM -0500, Bruce Momjian wrote: > On Tue, Jan 16, 2018 at 02:33:05PM +0900, Michael Paquier wrote: > > This bit is important. I am happy that your patch mentions that > > intermediate certificates avoid the need to store root ones on the > > client. Should the docs mention terms like "chain of trust"? > > I think the question is how much do we want to "teach" people in our > docs. We do oddly but wisely link from our docs to HP OpenVMS docs > about how the chain of trust works: > > http://h41379.www4.hpe.com/doc/83final/ba554_90007/ch04s02.html > > I will write up a paragraph about the concepts for our docs for the > group's review. As a separate patch, I think that it would be fine as well. > > Perhaps the docs could also include an example of command to create a > > root and an intermediate certificate in runtime.sgml or such? > > Yes, I have thought about that. My presentation has clear examples that > we can use, again based on Stephen and David's scripts using v3_ca. I > will work up a possible patch for that too. That too. > > On top of that, src/test/ssl does not provide any kind of coverage for > > that. It would be an area of improvement for those tests. > > Wow, I have no idea how to do that. Let me look. Seems I have more > work to do. You would need to update src/test/ssl/Makefile to generate those intermediate CAs, and then make ServerSetup::switch_server_cert smarter in the way the series of certificates are handled. A suggestion I have would be to create each certificate file separately and change the routine so as it uses an array in input, the order of the items defining what's the order the the data. For the client there is sslrootcert, so I guess that a small routine able to take a set of certs and append them to a single file would make it as well (switch_server_cert should use it). > Instead of appending to this doc patch, I will work on a second one for > review. I see nothing pressing here. If you are not familiar with the TAP test facility, this could give you a good introduction to it. -- Michael
Attachment
On Wed, Jan 17, 2018 at 09:09:50AM +0900, Michael Paquier wrote: > On Tue, Jan 16, 2018 at 11:21:22AM -0500, Bruce Momjian wrote: > > On Tue, Jan 16, 2018 at 02:33:05PM +0900, Michael Paquier wrote: > > > This bit is important. I am happy that your patch mentions that > > > intermediate certificates avoid the need to store root ones on the > > > client. Should the docs mention terms like "chain of trust"? > > > > I think the question is how much do we want to "teach" people in our > > docs. We do oddly but wisely link from our docs to HP OpenVMS docs > > about how the chain of trust works: > > > > http://h41379.www4.hpe.com/doc/83final/ba554_90007/ch04s02.html > > > > I will write up a paragraph about the concepts for our docs for the > > group's review. > > As a separate patch, I think that it would be fine as well. I ended up merging the "chain of trust" changes into the "intermediate" patch since they affect adjacent sections of the docs. You can see this as the first attached patch. > > > Perhaps the docs could also include an example of command to create a > > > root and an intermediate certificate in runtime.sgml or such? > > > > Yes, I have thought about that. My presentation has clear examples that > > we can use, again based on Stephen and David's scripts using v3_ca. I > > will work up a possible patch for that too. > > That too. I did that as a separate patch, which is the second attachment. > > > On top of that, src/test/ssl does not provide any kind of coverage for > > > that. It would be an area of improvement for those tests. > > > > Wow, I have no idea how to do that. Let me look. Seems I have more > > work to do. > > You would need to update src/test/ssl/Makefile to generate those > intermediate CAs, and then make ServerSetup::switch_server_cert smarter > in the way the series of certificates are handled. A suggestion I have > would be to create each certificate file separately and change the > routine so as it uses an array in input, the order of the items defining > what's the order the the data. For the client there is sslrootcert, so I > guess that a small routine able to take a set of certs and append them > to a single file would make it as well (switch_server_cert should use > it). I don't think I will work on the testing changes. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
Attachment
On Tue, Jan 16, 2018 at 10:23:44PM -0500, Bruce Momjian wrote: > On Wed, Jan 17, 2018 at 09:09:50AM +0900, Michael Paquier wrote: > > On Tue, Jan 16, 2018 at 11:21:22AM -0500, Bruce Momjian wrote: > > > On Tue, Jan 16, 2018 at 02:33:05PM +0900, Michael Paquier wrote: > > I ended up merging the "chain of trust" changes into the "intermediate" > patch since they affect adjacent sections of the docs. You can see this > as the first attached patch. Thanks. I looked at crt.diff and the surroundings in the docs. This one looks consistent to me. > > > > Perhaps the docs could also include an example of command to create a > > > > root and an intermediate certificate in runtime.sgml or such? > > > > > > Yes, I have thought about that. My presentation has clear examples that > > > we can use, again based on Stephen and David's scripts using v3_ca. I > > > will work up a possible patch for that too. > > > > That too. > > I did that as a separate patch, which is the second attachment. This is openssl.diff. + Then, sign the request with the the private key to create a root +certificate authority: s/the the/the/ +<programlisting> +openssl req -new -nodes -text -out root.csr \ + -keyout root.key -subj "/CN=<replaceable>root.yourdomain.com</replaceable>" +chmod og-rwx root.key +openssl x509 -req -in root.csr -text -days 365 \ + -extfile /etc/ssl/openssl.cnf -extensions v3_ca \ + -signkey root.key -out root.crt The succession of commands of commands for the intermediate certificates is wild. Could it be possible to explain what each command means? Users would not get lost this way. > I don't think I will work on the testing changes. Fine for me. This could do for a fine TODO item. Not one of those hard, complicated and basically impossible things on the TODO list. -- Michael
Attachment
On Wed, Jan 17, 2018 at 05:20:00PM +0900, Michael Paquier wrote: > On Tue, Jan 16, 2018 at 10:23:44PM -0500, Bruce Momjian wrote: > > On Wed, Jan 17, 2018 at 09:09:50AM +0900, Michael Paquier wrote: > > > On Tue, Jan 16, 2018 at 11:21:22AM -0500, Bruce Momjian wrote: > > > > On Tue, Jan 16, 2018 at 02:33:05PM +0900, Michael Paquier wrote: > > > > I ended up merging the "chain of trust" changes into the "intermediate" > > patch since they affect adjacent sections of the docs. You can see this > > as the first attached patch. > > Thanks. I looked at crt.diff and the surroundings in the docs. This one > looks consistent to me. Good, thanks. > > I did that as a separate patch, which is the second attachment. > > This is openssl.diff. > > + Then, sign the request with the the private key to create a root > +certificate authority: > s/the the/the/ > > +<programlisting> > +openssl req -new -nodes -text -out root.csr \ > + -keyout root.key -subj "/CN=<replaceable>root.yourdomain.com</replaceable>" > +chmod og-rwx root.key > +openssl x509 -req -in root.csr -text -days 365 \ > + -extfile /etc/ssl/openssl.cnf -extensions v3_ca \ > + -signkey root.key -out root.crt > The succession of commands of commands for the intermediate certificates > is wild. Could it be possible to explain what each command means? Users > would not get lost this way. Yes, I was not happy about that either. I was afraid that pound-sign comments would look like root prompts but I just added them and they look fine. Updated patch attached, with some expiration and wording adjustments. There is also a new paragraph at the end explaining where to place the files. > > I don't think I will work on the testing changes. > > Fine for me. This could do for a fine TODO item. Not one of those hard, > complicated and basically impossible things on the TODO list. Agreed. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
Attachment
On Wed, Jan 17, 2018 at 07:34:42AM -0500, Bruce Momjian wrote: > > The succession of commands of commands for the intermediate certificates > > is wild. Could it be possible to explain what each command means? Users > > would not get lost this way. > > Yes, I was not happy about that either. I was afraid that pound-sign > comments would look like root prompts but I just added them and they > look fine. Updated patch attached, with some expiration and wording > adjustments. There is also a new paragraph at the end explaining where > to place the files. Oh, and how far back should these be backpatched? 10? 9.6? 9.3? I am thinking it should be done as far back as possible as long as it is simple. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
On Wed, Jan 17, 2018 at 08:39:55AM -0500, Bruce Momjian wrote: > On Wed, Jan 17, 2018 at 07:34:42AM -0500, Bruce Momjian wrote: > > > The succession of commands of commands for the intermediate certificates > > > is wild. Could it be possible to explain what each command means? Users > > > would not get lost this way. > > > > Yes, I was not happy about that either. I was afraid that pound-sign > > comments would look like root prompts but I just added them and they > > look fine. Updated patch attached, with some expiration and wording > > adjustments. There is also a new paragraph at the end explaining where > > to place the files. > > Oh, and how far back should these be backpatched? 10? 9.6? 9.3? I am > thinking it should be done as far back as possible as long as it is > simple. No objections from here to back-patch depending on the difficulty. I think that a0572203 has created some conflicts in this area for REL_10_STABLE, nothing huge though. -- Michael
Attachment
On Wed, Jan 17, 2018 at 07:34:42AM -0500, Bruce Momjian wrote: > On Wed, Jan 17, 2018 at 05:20:00PM +0900, Michael Paquier wrote: > > The succession of commands of commands for the intermediate certificates > > is wild. Could it be possible to explain what each command means? Users > > would not get lost this way. > > Yes, I was not happy about that either. I was afraid that pound-sign > comments would look like root prompts but I just added them and they > look fine. Updated patch attached, with some expiration and wording > adjustments. There is also a new paragraph at the end explaining where > to place the files. Thanks, that's a net improvement. So +1 for this version. + enterprise-wide root <acronym>CAs</acronym>) should be used in production. Nit here. CA should not be plural. +</programlisting> + Then, sign the request with the the key to create a root certificate + authority: You still have a "the the" here. /etc/ssl/openssl.cnf is not available on macos or Windows, which can lead to a bit of confusion as I would imagine that people would copy/paste such commands when testing things. Perhaps it would be worth mentioning that this path is proper to usual Linux distributions (I can see it at least on ArchLinux and Debian), with a reference to this OpenSSL link: https://www.openssl.org/docs/manmaster/man5/config.html There is as well a set of tiny configuration files in src/test/ssl. -- Michael
Attachment
On Thu, Jan 18, 2018 at 10:25:03AM +0900, Michael Paquier wrote: > On Wed, Jan 17, 2018 at 07:34:42AM -0500, Bruce Momjian wrote: > > Yes, I was not happy about that either. I was afraid that pound-sign > > comments would look like root prompts but I just added them and they > > look fine. Updated patch attached, with some expiration and wording > > adjustments. There is also a new paragraph at the end explaining where > > to place the files. > > Thanks, that's a net improvement. So +1 for this version. > > + enterprise-wide root <acronym>CAs</acronym>) should be used in production. > Nit here. CA should not be plural. > > +</programlisting> > + Then, sign the request with the the key to create a root certificate > + authority: > You still have a "the the" here. > > /etc/ssl/openssl.cnf is not available on macos or Windows, which can > lead to a bit of confusion as I would imagine that people would > copy/paste such commands when testing things. Perhaps it would be worth > mentioning that this path is proper to usual Linux distributions (I can > see it at least on ArchLinux and Debian), with a reference to this > OpenSSL link: > https://www.openssl.org/docs/manmaster/man5/config.html > > There is as well a set of tiny configuration files in src/test/ssl. One odd thing about the configuration file is that you don't need to modify it, but you do need to specify it for that command. Fixed patch attached. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
Attachment
On Wed, Jan 17, 2018 at 09:00:17PM -0500, Bruce Momjian wrote: > On Thu, Jan 18, 2018 at 10:25:03AM +0900, Michael Paquier wrote: > > /etc/ssl/openssl.cnf is not available on macos or Windows, which can > > lead to a bit of confusion as I would imagine that people would > > copy/paste such commands when testing things. Perhaps it would be worth > > mentioning that this path is proper to usual Linux distributions (I can > > see it at least on ArchLinux and Debian), with a reference to this > > OpenSSL link: > > https://www.openssl.org/docs/manmaster/man5/config.html > > One odd thing about the configuration file is that you don't need to > modify it, but you do need to specify it for that command. > > Fixed patch attached. OK, what you are proposing here looks fine to me. That's an improvement. -- Michael
Attachment
On Thu, Jan 18, 2018 at 12:17:40PM +0900, Michael Paquier wrote: > On Wed, Jan 17, 2018 at 09:00:17PM -0500, Bruce Momjian wrote: > > On Thu, Jan 18, 2018 at 10:25:03AM +0900, Michael Paquier wrote: > > > /etc/ssl/openssl.cnf is not available on macos or Windows, which can > > > lead to a bit of confusion as I would imagine that people would > > > copy/paste such commands when testing things. Perhaps it would be worth > > > mentioning that this path is proper to usual Linux distributions (I can > > > see it at least on ArchLinux and Debian), with a reference to this > > > OpenSSL link: > > > https://www.openssl.org/docs/manmaster/man5/config.html > > > > One odd thing about the configuration file is that you don't need to > > modify it, but you do need to specify it for that command. > > > > Fixed patch attached. > > OK, what you are proposing here looks fine to me. That's an > improvement. Applied back through 9.3. Thanks for the review and suggestions. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
On 1/16/18 00:33, Michael Paquier wrote:
> On top of that, src/test/ssl does not provide any kind of coverage for
> that. It would be an area of improvement for those tests.
The tests already cover this:
# intermediate client_ca.crt is provided by client, and isn't in
server's ssl_ca_file
switch_server_cert($node, 'server-cn-only', 'root_ca');
$common_connstr =
"user=ssltestuser dbname=certdb sslkey=ssl/client_tmp.key
sslrootcert=ssl/root+server_ca.crt hostaddr=$SERVERHOSTADDR";
test_connect_ok($common_connstr,
"sslmode=require sslcert=ssl/client+client_ca.crt");
test_connect_fails($common_connstr, "sslmode=require
sslcert=ssl/client.crt");
If you change the Makefile rule for generating the client CA to omit the
-extensions v3_ca option, then the first test will fail.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Thu, Jan 25, 2018 at 10:59:23PM -0500, Peter Eisentraut wrote: > On 1/16/18 00:33, Michael Paquier wrote: > > On top of that, src/test/ssl does not provide any kind of coverage for > > that. It would be an area of improvement for those tests. > > The tests already cover this: > > # intermediate client_ca.crt is provided by client, and isn't in > server's ssl_ca_file > switch_server_cert($node, 'server-cn-only', 'root_ca'); > $common_connstr = > "user=ssltestuser dbname=certdb sslkey=ssl/client_tmp.key > sslrootcert=ssl/root+server_ca.crt hostaddr=$SERVERHOSTADDR"; > > test_connect_ok($common_connstr, > "sslmode=require sslcert=ssl/client+client_ca.crt"); > test_connect_fails($common_connstr, "sslmode=require > sslcert=ssl/client.crt"); > > If you change the Makefile rule for generating the client CA to omit the > -extensions v3_ca option, then the first test will fail. Oh, very good! -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + As you are, so once was I. As I am, so you will be. + + Ancient Roman grave inscription +
On Fri, Jan 26, 2018 at 08:09:30AM -0500, Bruce Momjian wrote: > On Thu, Jan 25, 2018 at 10:59:23PM -0500, Peter Eisentraut wrote: > > If you change the Makefile rule for generating the client CA to omit the > > -extensions v3_ca option, then the first test will fail. > > Oh, very good! Good point, I didn't notice that. Thanks Peter. -- Michael