Thread: PostgreSQL Trusted Startup

Hello,

I am investigating security requirements for configuring a PostgreSQL database on a Linux system.
One of the security requirements our organization would like to implement is "trusted startup", in that PostgreSQL would verify the authenticity of the binaries and configuration files before making the database available to users.  This would enable the database to detect if the system has possibly been compromised.
Since this is a Linux system, I could keep a list of known good MD5 checksums and compare the checksums prior to startup by editing the init script.  The list would of course need to be updated any time I make a configuration change or apply a patch.
Is there an alternative method of implementing such a requirement?  Possibly one already incorporated into PostgreSQL?

Thanks,

Ken Buckler

On Mon, Dec 20, 2010 at 12:12 PM, Kenneth Buckler
<kenneth.buckler@gmail.com> wrote:
> Hello,
>
> I am investigating security requirements for configuring a PostgreSQL
> database on a Linux system.
> One of the security requirements our organization would like to implement is
> "trusted startup", in that PostgreSQL would verify the authenticity of the
> binaries and configuration files before making the database available to
> users.  This would enable the database to detect if the system has possibly
> been compromised.

But, if the script is run on the same machine as postgresql is on, the
scripts that check for changes could be compromised as well and then
you'd never know.

> Since this is a Linux system, I could keep a list of known good MD5
> checksums and compare the checksums prior to startup by editing the init
> script.  The list would of course need to be updated any time I make a
> configuration change or apply a patch.
> Is there an alternative method of implementing such a requirement?  Possibly
> one already incorporated into PostgreSQL?

pgsql doesn't do any of that, but I'm sure you can roll your own so to
speak.  I would tend to write some kind of nagios plugin that could be
called remotely that would notify you whenever it changes so you would
know as soon as a change occurred rather than later when trying to
restart the database during a midday outage while the boss screams
"get the system back up now! We're losing money!"

Generally, if the db's been compromised, someone's already gotten to
an app server or two, and might be sniffing traffic anyway, so it's
likely a lost cause by then.

On 12/20/10 11:12 AM, Kenneth Buckler wrote:
> Hello,
>
> I am investigating security requirements for configuring a PostgreSQL
> database on a Linux system.
> One of the security requirements our organization would like to
> implement is "trusted startup", in that PostgreSQL would verify the
> authenticity of the binaries and configuration files before making the
> database available to users.  This would enable the database to detect
> if the system has possibly been compromised.
> Since this is a Linux system, I could keep a list of known good MD5
> checksums and compare the checksums prior to startup by editing the
> init script.  The list would of course need to be updated any time I
> make a configuration change or apply a patch.
> Is there an alternative method of implementing such a requirement?
> Possibly one already incorporated into PostgreSQL?

I would look into selinux.   lock it down with this, and it will be much
harder to compromise.

On Mon, Dec 20, 2010 at 1:43 PM, John R Pierce <pierce@hogranch.com> wrote:
> I would look into selinux.   lock it down with this, and it will be much
> harder to compromise.

I agree.  By the time you've got compromised binaries / config files
on the system, you've already lost.

On Mon, Dec 20, 2010 at 3:31 PM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
>
>
> But, if the script is run on the same machine as postgresql is on, the
> scripts that check for changes could be compromised as well and then
> you'd never know.
>

I agree, if the system has been compromised, nothing will prevent the
scripts from being compromised.
Hence why I am looking for alternatives.  I consider the md5 script
approach a poor approach, but it does meet the letter of the
requirements set forth by the organization.

> > Is there an alternative method of implementing such a requirement?  Possibly
> > one already incorporated into PostgreSQL?
>
> pgsql doesn't do any of that, but I'm sure you can roll your own so to
> speak.  I would tend to write some kind of nagios plugin that could be
> called remotely that would notify you whenever it changes so you would
> know as soon as a change occurred rather than later when trying to
> restart the database during a midday outage while the boss screams
> "get the system back up now! We're losing money!"
>

Thanks for clarifying that for me.  Part of the requirement I'm
working with requires "vendor documentation" stating if such a feature
exists.  Since there is no vendor documentation, they'll have to
settle for my own documentation, backed up with a mailing list post.

Writing my own plugin/module hasn't been ruled out.  I wanted to make
sure that I'm not re-inventing the wheel.

In any approach to this, I will be including an override which will
allow PostgreSQL to start despite failing the "trusted files" check.

> Generally, if the db's been compromised, someone's already gotten to
> an app server or two, and might be sniffing traffic anyway, so it's
> likely a lost cause by then.

Agreed.  Unfortunately, I've been given specific requirements and I am
obligated to fulfill those requirements, even if I don't agree those
requirements are necessary.  This is all in addition to an extensive
OS lockdown script, as well as additional lockdown requirements for
the database.

I appreciate the help.  I believe this is an excellent starting point
to try and address this requirement.

Ken

On 12/21/2010 06:12 AM, Kenneth Buckler wrote:
> Hello,
>
> I am investigating security requirements for configuring a PostgreSQL
> database on a Linux system.
> One of the security requirements our organization would like to
> implement is "trusted startup", in that PostgreSQL would verify the
> authenticity of the binaries and configuration files before making the
> database available to users.

Do you have a trusted boot path from BIOS to bootloader to kernel to
init core userspace, where everything is digitally signed (by you or
someone else) and verified before execution? Do you disable kernel
module loading?

If not, you're wasting your time, because a compromise by malicious
kernel module, modified init, modified md5 command, etc will render your
precautions totally pointless.

If your BIOS can't verify the bootloader, which is likely on an x86 /
x64 system, then you can still get some protection by signing your
kernels and using a bootloader that checks signatures. If someone messes
with the bootloader you lose, but it'll help protect you against obvious
automated attacks. You might be able to use the Trusted Platform Module
(TPM) on your machine to get a fully verified chain of trust, though, by
using Trusted GRUB.

http://trousers.sourceforge.net/grub.html

If you can reasonably trust that the kernel you loaded is OK, you can
have it verify signatures on binaries before executing them. There was a
DigSig project for that (http://disec.sourceforge.net/) but it seems to
have stopped recently. I'm not sure if there's any replacement.

Without kernel-level signature verification, all you can really do is
have a custom initrd/initramfs (signed and verified by grub during boot)
that checks the signatures on init, md5, gpg, libc, etc etc (any binary
root runs, including scripts) before switching to the real root FS
during boot. Then you can have your Pg startup scripts (which you signed
on a separate, trusted machine) verify GnuPG signatures of the Pg
binaries before execution.

All in all, it's a painful, clumsy way to do things, and AFAIK there's
little support in mainline Linux systems for trusted boot and
trusted-binary systems. You might find out more with a search for "linux
trusted computing", "linux trusted boot", "linux tpm", "linux signed
binaries", etc.

Personally, I'd be using existing system- and network-level intrusion
detection tools like tripwire and snort to try to spot intrusion if and
when it happens. I'm not confident that a chain-of-trust approach is
workable on Linux systems at present, though I'd love to be proved wrong
by being pointed at existing support I've missed.

--
Craig Ringer

On Mon, Dec 20, 2010 at 8:53 PM, Craig Ringer
<craig@postnewspapers.com.au> wrote:
>
> Do you have a trusted boot path from BIOS to bootloader to kernel to init
> core userspace, where everything is digitally signed (by you or someone
> else) and verified before execution? Do you disable kernel module loading?
>
> If not, you're wasting your time, because a compromise by malicious kernel
> module, modified init, modified md5 command, etc will render your
> precautions totally pointless.
>
> If your BIOS can't verify the bootloader, which is likely on an x86 / x64
> system, then you can still get some protection by signing your kernels and
> using a bootloader that checks signatures. If someone messes with the
> bootloader you lose, but it'll help protect you against obvious automated
> attacks. You might be able to use the Trusted Platform Module (TPM) on your
> machine to get a fully verified chain of trust, though, by using Trusted
> GRUB.
>
> http://trousers.sourceforge.net/grub.html
>
> If you can reasonably trust that the kernel you loaded is OK, you can have
> it verify signatures on binaries before executing them. There was a DigSig
> project for that (http://disec.sourceforge.net/) but it seems to have
> stopped recently. I'm not sure if there's any replacement.
>
> Without kernel-level signature verification, all you can really do is have a
> custom initrd/initramfs (signed and verified by grub during boot) that
> checks the signatures on init, md5, gpg, libc, etc etc (any binary root
> runs, including scripts) before switching to the real root FS during boot.
> Then you can have your Pg startup scripts (which you signed on a separate,
> trusted machine) verify GnuPG signatures of the Pg binaries before
> execution.
>
> All in all, it's a painful, clumsy way to do things, and AFAIK there's
> little support in mainline Linux systems for trusted boot and trusted-binary
> systems. You might find out more with a search for "linux trusted
> computing", "linux trusted boot", "linux tpm", "linux signed binaries", etc.
>
> Personally, I'd be using existing system- and network-level intrusion
> detection tools like tripwire and snort to try to spot intrusion if and when
> it happens. I'm not confident that a chain-of-trust approach is workable on
> Linux systems at present, though I'd love to be proved wrong by being
> pointed at existing support I've missed.
>
> --
> Craig Ringer
>

I find it very comforting that I am not the only one who finds this
requirement a bit "out there".
Unfortunately, these requirements are set in stone, and no matter how
hard I try, can not be altered.
We live in a world where compliance is king.  Nevermind if compliance
doesn't actually make the system more secure.

Unfortunately Tripwire does not meet the full requirement, as it does
not prevent the database from starting.

Ken

On 12/22/2010 02:05 AM, Kenneth Buckler wrote:

> I find it very comforting that I am not the only one who finds this
> requirement a bit "out there".
> Unfortunately, these requirements are set in stone, and no matter how
> hard I try, can not be altered.
> We live in a world where compliance is king.  Nevermind if compliance
> doesn't actually make the system more secure.
>
> Unfortunately Tripwire does not meet the full requirement, as it does
> not prevent the database from starting.

In this case, here's what I'd do:

- Call Red Hat
- Say "I'd like to buy RHEL, but have <x> weird requirement, can you
   help me"

--
Craig Ringer

> We live in a world where compliance is king.  Nevermind if compliance
> doesn't actually make the system more secure.

Er .. re my previous post, I don't mean "lie to RH and claim to want to
buy RHEL to get free support". I mean that you should consider going to
management and getting approval for professional support and integration
work from a specialist, because you're going to need it.

Alternately you could do the dodgy Trusted GRUB + signed kernel + signed
initrd with scripted GnuPG verification hack. It'd be a lot better than
nothing if your target server has a TPM you can enable and use for
Trusted GRUB.

--
Craig Ringer