Thread: [buildfarm-members] Moving to HTTPS

In common with the rest of the PostgreSQL infrastructure, we will
shortly be moving the buildfarm server to supporting HTTPS, and
eventually will stop support for plain HTTP. Many installations on
buildfarm animals will not need any great adjustment, others will need
some extra modules installed. Here's a simple test that should help tell
if your animal needs some extra stuff installed. Run something like this
on your animal, making sure that the perl in your path is the one used
to run the buildfarm client:


    perl -MLWP::Simple -e 'head(q{https://www.postgresql.org});


If this doesn't complain you should be good. If it does, you probably
need to install the LWP::Protocol::https module, and anything it depends
on (e.g. crypto libraries).

This isn't enabled yet, I'm just being proactive so you don't get caught
having to adjust in a hurry.

Windows users probably don't need to worry - this works on every
installation of AS perl I have.


cheers


andrew

On Sat, Jan 07, 2017 at 04:37:47PM -0500, Andrew Dunstan wrote:
> Here's a simple test that should help tell if your animal needs some extra
> stuff installed. Run something like this on your animal, making sure that
> the perl in your path is the one used to run the buildfarm client:
>
>
>    perl -MLWP::Simple -e 'head(q{https://www.postgresql.org});

I get no complaint from that, but I also get no complaint from these:

  perl -MLWP::Simple -e 'head(q{https://nonsense.postgresql.org});'; echo $?
  perl -MLWP::Simple -e 'head(q{nonsense://nonsense});'; echo $?

Maybe use this:

  perl -MData::Dumper -MLWP::Simple -e 'print Dumper [head(q{https://www.postgresql.org}) || "BAD"]'

On 01/07/2017 04:55 PM, Noah Misch wrote:
> On Sat, Jan 07, 2017 at 04:37:47PM -0500, Andrew Dunstan wrote:
>> Here's a simple test that should help tell if your animal needs some extra
>> stuff installed. Run something like this on your animal, making sure that
>> the perl in your path is the one used to run the buildfarm client:
>>
>>
>>     perl -MLWP::Simple -e 'head(q{https://www.postgresql.org});
> I get no complaint from that, but I also get no complaint from these:
>
>    perl -MLWP::Simple -e 'head(q{https://nonsense.postgresql.org});'; echo $?
>    perl -MLWP::Simple -e 'head(q{nonsense://nonsense});'; echo $?
>
> Maybe use this:
>
>    perl -MData::Dumper -MLWP::Simple -e 'print Dumper [head(q{https://www.postgresql.org}) || "BAD"]'
>


Yeah, you're right. Or perhaps even more simply we could do:

        perl -MLWP::Simple -e 'print qq{BAD\n} unless head(q{https://www.postgresql.org});'

cheers

andrew

--
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> Yeah, you're right. Or perhaps even more simply we could do:
>         perl -MLWP::Simple -e 'print qq{BAD\n} unless head(q{https://www.postgresql.org});'

Hmm, so that reports BAD on all four of my buildfarm critters :-(.

On longfin, which is running a current macOS release, "cpan install
LWP::Protocol::https" seems to be enough to fix it; but on the other
three, I get various flavors of failure.  It looks like the root cause is
that the versions of the relevant modules that are currently available
from CPAN assume newer versions of OpenSSL and/or other Perl modules than
these machines have installed.

I could embark on a massive update binge, but that would sort of defeat
the purpose of testing against old Perl versions.  Thoughts?

            regards, tom lane

On 01/07/2017 07:12 PM, Tom Lane wrote:
> Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
>> Yeah, you're right. Or perhaps even more simply we could do:
>>          perl -MLWP::Simple -e 'print qq{BAD\n} unless head(q{https://www.postgresql.org});'
> Hmm, so that reports BAD on all four of my buildfarm critters :-(.
>
> On longfin, which is running a current macOS release, "cpan install
> LWP::Protocol::https" seems to be enough to fix it; but on the other
> three, I get various flavors of failure.  It looks like the root cause is
> that the versions of the relevant modules that are currently available
> from CPAN assume newer versions of OpenSSL and/or other Perl modules than
> these machines have installed.
>
> I could embark on a massive update binge, but that would sort of defeat
> the purpose of testing against old Perl versions.  Thoughts?
>


Ugh. I was afraid of something like that. We might need to look at
providing a proxy or something.




cheers

andrew

--

Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Sat, Jan 07, 2017 at 07:12:07PM -0500, Tom Lane wrote:
> Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> > Yeah, you're right. Or perhaps even more simply we could do:
> >         perl -MLWP::Simple -e 'print qq{BAD\n} unless head(q{https://www.postgresql.org});'
>
> Hmm, so that reports BAD on all four of my buildfarm critters :-(.
>
> On longfin, which is running a current macOS release, "cpan install
> LWP::Protocol::https" seems to be enough to fix it; but on the other
> three, I get various flavors of failure.  It looks like the root cause is
> that the versions of the relevant modules that are currently available
> from CPAN assume newer versions of OpenSSL and/or other Perl modules than
> these machines have installed.
>
> I could embark on a massive update binge, but that would sort of defeat
> the purpose of testing against old Perl versions.  Thoughts?

You could use install a more modern Perl version just for the
buildfarm client while keeping the system perl marooned however far in
the past we promise to support.  Don't know just how painful that is
on those machines...

Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On 01/07/2017 07:12 PM, Tom Lane wrote:

Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:

Yeah, you're right. Or perhaps even more simply we could do:
        perl -MLWP::Simple -e 'print qq{BAD\n} unless head(q{https://www.postgresql.org});'

Hmm, so that reports BAD on all four of my buildfarm critters :-(.

On longfin, which is running a current macOS release, "cpan install
LWP::Protocol::https" seems to be enough to fix it; but on the other
three, I get various flavors of failure.  It looks like the root cause is
that the versions of the relevant modules that are currently available
from CPAN assume newer versions of OpenSSL and/or other Perl modules than
these machines have installed.

I could embark on a massive update binge, but that would sort of defeat
the purpose of testing against old Perl versions.  Thoughts?

Ugh. I was afraid of something like that. We might need to look at providing a proxy or something.

David Fetter <david@fetter.org> writes:
> On Sat, Jan 07, 2017 at 07:12:07PM -0500, Tom Lane wrote:
>> I could embark on a massive update binge, but that would sort of defeat
>> the purpose of testing against old Perl versions.  Thoughts?

> You could use install a more modern Perl version just for the
> buildfarm client while keeping the system perl marooned however far in
> the past we promise to support.  Don't know just how painful that is
> on those machines...

Yeah, I was thinking about the same: build a modern perl somewhere off to
the side (not in the normal PATH) and run the buildfarm script using that
while the build per se continues to use the system Perl.  The pain here
is going to mostly come from the buildfarm script itself, I'm afraid.
One thing I noticed right away is it's got /usr/bin/perl hardwired into
all its shebang lines.  Don't know what other gotchas there are.

            regards, tom lane

On 01/08/2017 10:34 AM, Tom Lane wrote:
> David Fetter <david@fetter.org> writes:
>> On Sat, Jan 07, 2017 at 07:12:07PM -0500, Tom Lane wrote:
>>> I could embark on a massive update binge, but that would sort of defeat
>>> the purpose of testing against old Perl versions.  Thoughts?
>> You could use install a more modern Perl version just for the
>> buildfarm client while keeping the system perl marooned however far in
>> the past we promise to support.  Don't know just how painful that is
>> on those machines...
> Yeah, I was thinking about the same: build a modern perl somewhere off to
> the side (not in the normal PATH) and run the buildfarm script using that
> while the build per se continues to use the system Perl.  The pain here
> is going to mostly come from the buildfarm script itself, I'm afraid.
> One thing I noticed right away is it's got /usr/bin/perl hardwired into
> all its shebang lines.  Don't know what other gotchas there are.
>
>



That's more or less how the Msys stuff works. The buildfarm scripts run
using the Msys DTK client (so we get the virtual paths right) but builds
against AS perl.

And in fact there is provision for running the web transaction using a
different perl. Right now it's only used in very limited circumstances
(older Msys where LWP isn't supported at all in the DTK perl) but we
could expand it. For example, we could run it any time aux_path is
actually set. See the code starting around line 2097 of run_build.pl.

There's no great urgency about this. And I want to do it with minimal
disturbance, so let's get it right.

cheers

andrew

Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> On 01/07/2017 07:12 PM, Tom Lane wrote:
>> Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
>>> Yeah, you're right. Or perhaps even more simply we could do:
>>> perl -MLWP::Simple -e 'print qq{BAD\n} unless head(q{https://www.postgresql.org});'

>> Hmm, so that reports BAD on all four of my buildfarm critters :-(.

> Ugh. I was afraid of something like that. We might need to look at
> providing a proxy or something.

After further fooling with this, it seems the root cause is that you
need Perl (specifically Net::SSLeay) to be linked to OpenSSL 0.9.8
or later.  Since we require at least 0.9.8 for PG itself, this does
not seem like an unreasonably heavy lift.

I found that some of the involved modules, particularly IO::Socket::SSL
and LWP::Protocol::https, fail a few of their regression tests even
with 0.9.8.  But if you tell cpan to ignore that and install them
anyway, it seems like things work; at least, the simple test proposed
above now works on all four of my buildfarm critters.  It will be
interesting to see whether that equates to "buildfarm script can talk
to https server".  Do we have any more-thorough test case available?

BTW, if anyone else is like me and has a more modern openssl installed
in a non-default location on an old machine, the secret sauce for getting
Net::SSLeay to use that is to set environment variable OPENSSL_PREFIX
to the openssl install prefix path before building Net::SSLeay.

            regards, tom lane

On 01/15/2017 03:39 AM, Tom Lane wrote:
> Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
>> On 01/07/2017 07:12 PM, Tom Lane wrote:
>>> Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
>>>> Yeah, you're right. Or perhaps even more simply we could do:
>>>> perl -MLWP::Simple -e 'print qq{BAD\n} unless head(q{https://www.postgresql.org});'
>>> Hmm, so that reports BAD on all four of my buildfarm critters :-(.
>> Ugh. I was afraid of something like that. We might need to look at
>> providing a proxy or something.
> After further fooling with this, it seems the root cause is that you
> need Perl (specifically Net::SSLeay) to be linked to OpenSSL 0.9.8
> or later.  Since we require at least 0.9.8 for PG itself, this does
> not seem like an unreasonably heavy lift.
>
> I found that some of the involved modules, particularly IO::Socket::SSL
> and LWP::Protocol::https, fail a few of their regression tests even
> with 0.9.8.  But if you tell cpan to ignore that and install them
> anyway, it seems like things work; at least, the simple test proposed
> above now works on all four of my buildfarm critters.  It will be
> interesting to see whether that equates to "buildfarm script can talk
> to https server".  Do we have any more-thorough test case available?



The test should be thorough enough. If you want to try it for real and
you're using the same that you built against, just change the URLs in
your config file, i.e.:

     sed -i -e
s,http://www.pgbuildfarm.org/,https://buildfarm.postgresql.org/,
your-config-file


cheers

andrew


--
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2017-01-15 14:59, Andrew Dunstan wrote:

> The test should be thorough enough. If you want to try it for real and
> you're using the same that you built against, just change the URLs in
> your config file, i.e.:
>
>     sed -i -e
> s,http://www.pgbuildfarm.org/,https://buildfarm.postgresql.org/,
> your-config-file

I tried to surf to:

https://pgbuildfarm.org/

and I am getting that the certificate is invalid.  I see that you are
using let's encrypt service for the SSL-certificate.  Just configure it
to include all the DNS-names that you are using.   Currently the
installed certificate is configured for the DNS-name:

buildfarm.postgresql.org

and with the SAN:

DNS Name: brentalia.postgresql.org
DNS Name: buildfarm.postgresql.org

I would suggest adding pgbuildfarm.org, www.pgbuildfarm.org etc also as
SAN:s.

It's a pity that they don't support wildcard-certs though.

/Mikael

Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> On 01/15/2017 03:39 AM, Tom Lane wrote:
>> ... it seems like things work; at least, the simple test proposed
>> above now works on all four of my buildfarm critters.  It will be
>> interesting to see whether that equates to "buildfarm script can talk
>> to https server".  Do we have any more-thorough test case available?

> The test should be thorough enough. If you want to try it for real and
> you're using the same that you built against, just change the URLs in
> your config file, i.e.:

>      sed -i -e
> s,http://www.pgbuildfarm.org/,https://buildfarm.postgresql.org/,
> your-config-file

Done, and all of my machines have successfully reported builds using
this URL.  I'll leave them like that unless you say I shouldn't.

            regards, tom lane

On 01/15/2017 03:09 PM, Mikael Kjellström wrote:
> On 2017-01-15 14:59, Andrew Dunstan wrote:
>
>> The test should be thorough enough. If you want to try it for real and
>> you're using the same that you built against, just change the URLs in
>> your config file, i.e.:
>>
>>     sed -i -e
>> s,http://www.pgbuildfarm.org/,https://buildfarm.postgresql.org/,
>> your-config-file
>
> I tried to surf to:
>
> https://pgbuildfarm.org/
>
> and I am getting that the certificate is invalid.  I see that you are
> using let's encrypt service for the SSL-certificate.  Just configure it
> to include all the DNS-names that you are using.   Currently the
> installed certificate is configured for the DNS-name:
>
> buildfarm.postgresql.org
>
> and with the SAN:
>
> DNS Name: brentalia.postgresql.org
> DNS Name: buildfarm.postgresql.org
>
> I would suggest adding pgbuildfarm.org, www.pgbuildfarm.org etc also as
> SAN:s.

Thanks for the suggestion, but that is not easily doable in our current
infrastructure - our letsencrypt deployment is fully automated and
completely integrated with our internal config management and nameserver
infrastructure and therefor only works with domains we also host and
manage through/on pginfra.




regards


Stefan

On 2017-01-16 10:24, Stefan Kaltenbrunner wrote:

>> I would suggest adding pgbuildfarm.org, www.pgbuildfarm.org etc also as
>> SAN:s.
>
> Thanks for the suggestion, but that is not easily doable in our current
> infrastructure - our letsencrypt deployment is fully automated and
> completely integrated with our internal config management and nameserver
> infrastructure and therefor only works with domains we also host and
> manage through/on pginfra.

Ok, I see.   It was only a suggestion.  Maybe it isn't that important to
use HTTPS on the pgbuildfarm-domain.

/Mikael