Thread: PG in cash till machines

PG in cash till machines

From
Carlos Henrique Reimer
Date:
Hi,

We are developing a solution which will run in thousands of small cash till machines running Linux and we would like to use PostgreSQL but there is a insecurity feeling regarding the solution basically because these boxes would be exposed to an insecure environment and insecure procedures like: non controlled power outages, untrusted hardware, no appropriate environment for hardware (dusty, hot) etc...

Our major concern is related to the write-back issues we can face in this environment. Is there a way to prevent it or a way PG can be configured to detect write-back configurations? Don't think so, but...

Basically, the question is: is there a way to guaranty a reliable PG database in an untrusted and insecure environment?  Maybe some kind of file system could give this guaranty, not sure..

--
Reimer

Re: PG in cash till machines

From
Merlin Moncure
Date:
On Fri, May 10, 2013 at 8:43 AM, Carlos Henrique Reimer
<carlos.reimer@opendb.com.br> wrote:
> Hi,
>
> We are developing a solution which will run in thousands of small cash till
> machines running Linux and we would like to use PostgreSQL but there is a
> insecurity feeling regarding the solution basically because these boxes
> would be exposed to an insecure environment and insecure procedures like:
> non controlled power outages, untrusted hardware, no appropriate environment
> for hardware (dusty, hot) etc...
>
> Our major concern is related to the write-back issues we can face in this
> environment. Is there a way to prevent it or a way PG can be configured to
> detect write-back configurations? Don't think so, but...

This is not the job of postgres, but of the operating system (and
possibly various hardware attached to the server).  In practice, it's
very possible to configure things so that a full flush all the way to
physical storage is guaranteed -- in fact that's the default behavior
for many configurations.  So this part, at least, is trivially done.

> Basically, the question is: is there a way to guaranty a reliable PG
> database in an untrusted and insecure environment?  Maybe some kind of file
> system could give this guaranty, not sure..

Again, this has more to do with the hardware (especially) operating
environment and the operating system than postgres itself.

merlin


Re: PG in cash till machines

From
Paul Jungwirth
Date:
>> Our major concern is related to the write-back issues we can face in this
>> environment.

I agree this is the OS's responsibility. Greg Smith's Postgres: High
Performance book has a lot to say about this, but there's also a lot
you could read online, since it's really a requirement for any
Postgres installation anywhere.

Paul

--
_________________________________
Pulchritudo splendor veritatis.


Re: PG in cash till machines

From
Bexley Hall
Date:
Hi Carlos,

On 5/10/2013 6:43 AM, Carlos Henrique Reimer wrote:
> We are developing a solution which will run in thousands of small cash till
> machines running Linux and we would like to use PostgreSQL but there is a
> insecurity feeling regarding the solution basically because these boxes
> would be exposed to an insecure environment and insecure procedures like:
> non controlled power outages, untrusted hardware, no appropriate
> environment for hardware (dusty, hot) etc...

Welcome to my world!  :>

To clarify your requirements:
- the devices contain hard currency which they either *accept* from
the user, dispense *to* the user, or both.
- the contents of the DB are critical to the proper operation
of the device (e.g., perhaps they maintain "account balances"
that, if corruptible, represent an exploit)
- presumably, the environment is secure enough that an adversary
can't just remove the entire device to abscound with the monies therein
(i.e., while it may not be "as secure as a bank lobby", neither is it
sitting in the middle of an open field!  Perhaps sitting *in* a
storefront establishment?)
- the device has some physical safeguards to prevent an adversary
vandalizing it GIVEN THE LEVEL OF SUPERVISION IN ITS ENVIRONMENT
(e.g., the coinbox of a US "pay phone" is pretty hard to break into)
- yet, a subtle, clever hacker could "tease" the device into a
state that is more "generous" to his wishes  (e.g., a high voltage
discharge at a key point along the case/wiring to crash the processor
at a critical point in it's transaction processing)
- a single device may not be a significant "value" target -- but,
an exploit that can be reapplied to *each* target (possibly
repeatedly, over time) makes this a sufficiently large honey pot
- you're very confident in the robustness of the underlying
OS, application and PostgreSQL itself that you don't fear problems
from any of those in "normal operation" (e.g., the kernel isn't
likely to panic due to some unforseen out-of-memory condition  -- nor
the processor hiccup from operating in an overtemperature condition)

> Our major concern is related to the write-back issues we can face in this
> environment. Is there a way to prevent it or a way PG can be configured to
> detect write-back configurations? Don't think so, but...

I'm not sure what specific threat you are anticipating, here.
Could you posit a potential attack scenario against which you
would like to defend?

> Basically, the question is: is there a way to guaranty a reliable PG
> database in an untrusted and insecure environment?  Maybe some kind of file
> system could give this guaranty, not sure..

Absent *physical* security, you can't do *anything*!  (especially
given your reliance on FOSS components!).  E.g., an adversary can
disembowel the device and massage the contents of the raw media,
etc.)

Having designed (regulated) gaming and "grey area" devices (each
handling hard currency), I can tell you that you have to have already
performed a pretty exhaustive threat analysis (e.g., red team, blue\
team) *before* you start the product's design.  If you can't imagine
*all* of the ways you can be targeted, then you can't determine
how/if you will be "secure" in each of those scenarios (e.g.,
I've incorporated features into the hardware designs to counter
certain types of physical attacks).

Always consider how you can design the system to *screw* the
attacker instead of *reward* him, given the option.  E.g.,
given the (fictitious) example of allowing the user to
"withdraw" funds from his "account" (possibly), then opt
for:
      decrease_balance(amount)
      dispense_currency(amount)
instead of:
      dispense_currency(amount)
      decrease_balance(amount)
as the latter case could allow an attacker to interrupt the
atomic operation *after* receiving his "hard currency" yet
before a record of this has been made.  In the former case,
an attack results in his account *losing* money without ever
receiving the hard currency!

Attackers are shy about complaining that "the machine cheated
me as that brings attention to their activities!  ("Oh?  Let's
review the video surveillance to see what happened..."  :>)

Of course, the above *should* be atomic but you can't roll-back
a *physical* transaction (i.e., reach out and grab the currency
*from* him if the transaction is aborted).

Perhaps if you can highlight specific actions that you imagine
as being vulnerabilities, a compromise solution might be
available?

[BTW, I find these the most delightful projects as you *really*
have to think outside the box -- unlike a desktop/server environment
where all you have to worry about is a disk crash, DoS attack, etc.]

G'luck!
--don


Re: PG in cash till machines

From
John R Pierce
Date:
On 5/10/2013 2:11 PM, Bexley Hall wrote:
Having designed (regulated) gaming and "grey area" devices (each
handling hard currency), I can tell you that you have to have already
performed a pretty exhaustive threat analysis (e.g., red team, blue\
team) *before* you start the product's design.  If you can't imagine
*all* of the ways you can be targeted, then you can't determine
how/if you will be "secure" in each of those scenarios (e.g.,
I've incorporated features into the hardware designs to counter
certain types of physical attacks).

indeed, and there's always threat models that no one could foresee, witness the recent story of coordinated ATM withdrawals of $45,000,000 enabled by some back door hacking of the bank databases.



-- 
john r pierce                                      37N 122W
somewhere on the middle of the left coast

Re: PG in cash till machines

From
Bexley Hall
Date:
Hi John,

On 5/10/2013 2:55 PM, John R Pierce wrote:
> On 5/10/2013 2:11 PM, Bexley Hall wrote:
>> Having designed (regulated) gaming and "grey area" devices (each
>> handling hard currency), I can tell you that you have to have already
>> performed a pretty exhaustive threat analysis (e.g., red team, blue\
>> team) *before* you start the product's design. If you can't imagine
>> *all* of the ways you can be targeted, then you can't determine
>> how/if you will be "secure" in each of those scenarios (e.g.,
>> I've incorporated features into the hardware designs to counter
>> certain types of physical attacks).
>
> indeed, and there's always threat models that no one could foresee,
> witness the recent story of coordinated ATM withdrawals of $45,000,000
> enabled by some back door hacking of the bank databases.

All (?) software and security flaws can be traced to "improper
assumptions".  Someone, somewhere, involved in the design of the
"system" (which includes more than just hardware and software)
made some BASIC assumption that was flawed.

I.e., assumed "/* CAN'T HAPPEN */" actually *couldn't* happen!

The trick to designing robust software (and robust systems) is
to identify those assumptions and then seriously question whether
they *really* are valid -- or, just valid in your particular
outlook on Life, The Universe and Everything.

- This instruction got executed so the next one will, as well.
- The only way for this instruction to be executed is if the
   preceeding one is, also.
- There can't be two credit cards with the same account number.
- People can't be in geographically different locations at the
   same time (so they can't possibly make withdrawals on the
   same account from those different locations, concurrently)
- Social security numbers can't begin with '0'.
- System power won't fail (or be intentionally interrupted)
   before I get a chance to do X... (e.g., flush buffers to
   permanent storage)
- All first names are less than BUFLEN characters.
- When turning *right*, I can't hit anything on the *left*.
- No one will unplug (or cut!) this cable.
- Users will choose "good" passwords.
- malloc() will never FAIL.
- This contact will close each time a coin is dispensed.
- Coins can't pass this sensor faster than once every N seconds.
etc.

Conversely, the way to find faults in those systems/software is
to "do the unexpected".  Developers tend to be *so* focused on
"just trying to get it to work" that they often don't attend to
"getting it to work *well*".

Carlos has to figure out which of these assumptions he's made
that aren't *guaranteed* (by some agency/mechanism) to be true
and figure out how to *make* them true (or, detect when they
are not).

--don


Re: PG in cash till machines

From
Chris Travers
Date:
Having some experience in the point of sale area, you are right to be concerned.  Before I go to inline points, I just want to say that my experience is that PostgreSQL does an admirable job reliability-wise in such environments, but it isn't perfect (no software solution can be) and the problems that remain, quite frankly, are not ones you can completely solve so a part of the strategy needs to be containment of damage.


On Fri, May 10, 2013 at 6:43 AM, Carlos Henrique Reimer <carlos.reimer@opendb.com.br> wrote:
Hi,

We are developing a solution which will run in thousands of small cash till machines running Linux and we would like to use PostgreSQL but there is a insecurity feeling regarding the solution basically because these boxes would be exposed to an insecure environment and insecure procedures like: non controlled power outages, untrusted hardware, no appropriate environment for hardware (dusty, hot) etc...

PostgreSQL isn't perfect.  Of course no software will be perfect with possibly bad electrical inputs and overheating CPU's.... 

Additionally my experience working with point of sale systems is that customers *will* abuse the systems.  I had one customer call me about database corruption (turned out to only be index corruption thankfully) and I showed up to find they had put the database server up in an unventillated closet above door level, so it was sitting in a cloud of its own hot air....  Especially when customers don't understand computer they will do things that will cause problems and you can plan on it.

However, my experience has been that even on desktop-level hardware PostgreSQL has been more reliable in such environments than any other database I have seen used in small business point of sale systems.  Despite horrid abuse of hardware the worst I ever saw was index corruption.  Some other systems caused problems that required periodic restoration from backup and loss of at least a day's data.

However you really want to plan for the possibility of losing a day's data.  Fortunately the customers who most abuse their hardware tend to be the most tolerant to such data loss. 


Our major concern is related to the write-back issues we can face in this environment. Is there a way to prevent it or a way PG can be configured to detect write-back configurations? Don't think so, but...

With a little development effort (C-language UDF's) you can make PostgreSQL detect anything you can detect through programming applications in the system.  You can then hook such detection functions into your app.  However, if the app is running on the same system, it may be easier to just do the detection in the application itself.

Basically, the question is: is there a way to guaranty a reliable PG database in an untrusted and insecure environment?  Maybe some kind of file system could give this guaranty, not sure..


To some extent, yes.  However the filesystem can only do so much if, for example, hard drives lie to the filesystem.....

Anyway, here is my recommendation:

1.  Plan on having anyone running multiple tills to hit a centralized server.  That makes it much easier to centralize these guarantees.  

2.  Document possible issues  for those implementing the system.  Discuss them on both a hardware and software level.

3.  Rely on implementors to do primary database support and implementation.  This hopefully allows some reasonable feedback and local knowledge even if the end user may not know how to keep things running effectively. 

4.  I would suggest building a pg_base_backup run into the daily closing so that if you need to restore from a backup this can be done quickly and easily.  This would be in addition to off-site backups (or even off-machine backups).

Basically rather than worrying about the guarantees from a technical level, I would be looking at it from a human angle first, and then technical guarantees for containment if something goes horribly wrong.  Your best technical measures may depend on things you can't depend on.  For example if the CPU overheats maybe send a message to someone urging technical support.... but what if the internet connection is down?

It wouldn't be a bad idea to have a diagnostic tool which could detect possible issues and which could be run demand (perhaps over the phone) but don't underestimate the importance of human contact, particularly on-site, in these sorts of environments.  A person visiting the site will spot issues that your software hasn't thought of and so it's a good idea to be thinking in terms of supplementing that.

Best Wishes,
Chris Travers

--
Reimer