Thread: Re: [HACKERS] Krb5 & multiple DB connections

Greetings,

  The attached patch fixes a bug which was originally brought up in May
  of 2002 in this thread:

  http://archives.postgresql.org/pgsql-interfaces/2002-05/msg00083.php

  The original bug reporter also supplied a patch to fix the problem:

  http://archives.postgresql.org/pgsql-interfaces/2002-05/msg00090.php

  I ran into exactly the same issue using 8.1.2.  I've updated the
  original patch to work for HEAD and 8.1.2.  I've tested the patch on
  both HEAD and 8.1.2, both at home and at work, and it works quite
  nicely.  In fact, I hope to have a patch to phppgadmin which will make
  it properly handle Kerberized logins.

      Thanks,

        Stephen

Stephen Frost <sfrost@snowman.net> writes:
>   The attached patch fixes a bug which was originally brought up in May
>   of 2002 in this thread:

Now that I've looked at it, I find this patch seems fairly wrongheaded.
AFAICS the entire point of the original coding is to allow the setup
work needed to create the krb5_context etc to be amortized across
multiple connections.  The patch destroys that advantage, but yet keeps
as much as it can of the notational cruft induced by the original
design -- for instance, there seems little point in the
pg_krb5_initialised flag if we aren't ever going to have any
pre-initialized state.

I have little idea of how expensive the operations called by
pg_krb5_init really are.  If they are expensive then it'd probably
make sense to keep the current static variables but treat 'em as a
one-element cache, ie, recompute if a new user name is being demanded.
If not, we ought to be able to simplify some things.

Another point here is how all this interacts with thread safety.
If we get rid of the static variables, do we still need the
pglock_thread() operations?

            regards, tom lane

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> >   The attached patch fixes a bug which was originally brought up in May
> >   of 2002 in this thread:
>
> Now that I've looked at it, I find this patch seems fairly wrongheaded.
> AFAICS the entire point of the original coding is to allow the setup
> work needed to create the krb5_context etc to be amortized across
> multiple connections.  The patch destroys that advantage, but yet keeps
> as much as it can of the notational cruft induced by the original
> design -- for instance, there seems little point in the
> pg_krb5_initialised flag if we aren't ever going to have any
> pre-initialized state.

I'm honestly not entirely sure I agree about that being the point of the
original coding but regardless the crux of the problem here is that
there's no way to get libpq to use a cache other than the one it's
initialized with for a given session.  That part of the Kerberos API
which supports that isn't exposed in any way beyond the KRB5CCNAME
environment variable and the call to ask for the 'default' ccache is
called with the static variable the second time and it ignores the
request when there's an already valid (it thinks) ccache.

> I have little idea of how expensive the operations called by
> pg_krb5_init really are.  If they are expensive then it'd probably
> make sense to keep the current static variables but treat 'em as a
> one-element cache, ie, recompute if a new user name is being demanded.
> If not, we ought to be able to simplify some things.

We'd have to recompute based on the KRB5CCNAME environment variable
changing, which is certainly an option.  It's not necessairly the case
that the username is changing, possibly just the cache.  Additionally,
the calls themselves are not very expensive when being called on an
existing cache, the most expensive thing is reaching out to the KDC to
get a new service ticket which will either need to be done, or won't,
depending on if a valid service ticket already exists in the cache or
not.

> Another point here is how all this interacts with thread safety.
> If we get rid of the static variables, do we still need the
> pglock_thread() operations?

Good question, I'm afraid probably not.  I'd have to look through it
again but last I checked MIT Kerberos prior to 1.4 (and I'm not 100%
sure it's resolved in 1.4) wasn't threadsafe itself.

I'd certainly be happy to rework the patch based on these comments, of
course.  Honestly, I'm pretty sure the original patch was intended to be
minimal (and is for the most part).  These changes would introduce more
logic but if that's alright I'd be happy to do it.

    Thanks,

        Stephen

Stephen Frost <sfrost@snowman.net> writes:
>> I have little idea of how expensive the operations called by
>> pg_krb5_init really are.  If they are expensive then it'd probably
>> make sense to keep the current static variables but treat 'em as a
>> one-element cache, ie, recompute if a new user name is being demanded.
>> If not, we ought to be able to simplify some things.

> We'd have to recompute based on the KRB5CCNAME environment variable
> changing, which is certainly an option.  It's not necessairly the case
> that the username is changing, possibly just the cache.

Hm, apparently I completely misunderstand the problem here.  What I
thought the bug was was that the cache wasn't recomputed given an
attempt to connect as a different Postgres username than the first
time.  If that's not the issue, then what is?

            regards, tom lane

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> >> I have little idea of how expensive the operations called by
> >> pg_krb5_init really are.  If they are expensive then it'd probably
> >> make sense to keep the current static variables but treat 'em as a
> >> one-element cache, ie, recompute if a new user name is being demanded.
> >> If not, we ought to be able to simplify some things.
>
> > We'd have to recompute based on the KRB5CCNAME environment variable
> > changing, which is certainly an option.  It's not necessairly the case
> > that the username is changing, possibly just the cache.
>
> Hm, apparently I completely misunderstand the problem here.  What I
> thought the bug was was that the cache wasn't recomputed given an
> attempt to connect as a different Postgres username than the first
> time.  If that's not the issue, then what is?

The specific problem which I and the original reporter ran into is this:

KRB5CCACHE=/tmp/krb5cc_apache_aev0kF
pg_connect() -- works fine
pg_close() -- works fine
rm /tmp/krb5cc_apache_aev0kF
KRB5CCACHE=/tmp/krb5cc_apache_cVMRtA
pg_connect() -- Doesn't work, Kerberos error is "no credentials cache"

What's happening here is that for every connection to apache by the
client a new credentials cache is created, and then destroyed when the
connection closes.  When using PHP (ie: phppgadmin) and mod_php (as is
common) the Apache process is the one actually making the connection to
the database and a given Apache process usually serves multiple requests
in its lifetime, sometimes to the same user, sometimes to different
users.

The static variables being used are for: krb5_init_context,
krb5_cc_default, krb5_cc_get_principal, and krb5_unparse_name.
Technically, between one connection and the next, krb5_cc_default,
krb5_cc_get_principal and krb5_unparse_name could reasonably return
different values.  krb5_init_context is pretty unlikely to change as
that would mean /etc/krb5.conf changed.  Looking through the krb5 source
code it appears that the only one which checks for something existing
is krb5_cc_default_name (called by krb5_cc_default), which will just
return the current ccache name if one has been set.
(src/lib/krb5/os/ccdefname.c:278, krb5-1.4.3)

We initially brought up this issue with the Kerberos folks actually:
http://pch.mit.edu/pipermail/kerberos/2006-February/009225.html

They pretty clearly felt that the application was responsible for
handling the cache in the event it changes.  Unfortunately, it's not the
application which is talking to Kerberos but another library in this
case which doesn't expose the Kerberos API to the application in such a
way to allow the application to notify Kerberos of the cache change.

Looking through the Kerberos API again it looks like it might be
possible to use krb5_cc_set_default_name(context, NULL) to force
krb5_cc_default_name() (from krb5_cc_default()) to re-find the cache.
Finding the cache again is reasonably inexpensive.  I've tried a couple
of things along these lines now but I havn't found a workable solution
which doesn't reinitialize the main Kerberos context, unfortunately.
I'll keep working on it as time allows though honestly I don't believe
it's generally terribly expensive to reinitialize the context...

Sorry it took so long to reply, running down the paths through the
various libraries takes a bit of time and I was really hoping to be able
to suggest an alternative solution using krb5_cc_set_default_name.

    Thanks,

        Stephen

Stephen Frost <sfrost@snowman.net> writes:
> The specific problem which I and the original reporter ran into is this:

> KRB5CCACHE=/tmp/krb5cc_apache_aev0kF
> pg_connect() -- works fine
> pg_close() -- works fine
> rm /tmp/krb5cc_apache_aev0kF
> KRB5CCACHE=/tmp/krb5cc_apache_cVMRtA
> pg_connect() -- Doesn't work, Kerberos error is "no credentials cache"

And why exactly is the application trying to munge the cache like that?
(I assume there is some state change you're not bothering to mention,
or there would be no need for a new cache, no?)

It sure seems like somebody is solving the wrong problem, and doing it
with a sledgehammer...

            regards, tom lane

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > The specific problem which I and the original reporter ran into is this:
>
> > KRB5CCACHE=/tmp/krb5cc_apache_aev0kF
> > pg_connect() -- works fine
> > pg_close() -- works fine
> > rm /tmp/krb5cc_apache_aev0kF
> > KRB5CCACHE=/tmp/krb5cc_apache_cVMRtA
> > pg_connect() -- Doesn't work, Kerberos error is "no credentials cache"
>
> And why exactly is the application trying to munge the cache like that?
> (I assume there is some state change you're not bothering to mention,
> or there would be no need for a new cache, no?)

Thought I mentioned it further down- each new Apache TCP connection gets
a new ccache file because it goes through the Negotiate protocol again
and gets a new TGT on the web server.

It works the exact same way as SSH, really.  When you connect to a
remote server using SSH it will store the TGT (assuming you have
delegate_credentials enabled) in a random file in /tmp.  It's not an
issue with SSH though because you don't tend to have applications which
continue running across multiple SSH sessions and which use the TGT to
reconnect to other services multiple times.

Actually, thinking about this a minute longer, it can happen w/
SSH/screen and psql.  Here's an example:

----------------------
sfrost@snowman:/data/sfrost/postgres/testinstall.krb5> kinit
Password for sfrost@SNOWMAN.NET:
sfrost@snowman:/data/sfrost/postgres/testinstall.krb5> ssh snowman
sfrost@snowman:/home/sfrost> screen
sfrost@snowman:/home/sfrost> psql -d template1 -h snowman
Welcome to psql 8.1.2, the PostgreSQL interactive terminal.
[...]
Disconnect from screen, log out, reconnect, reconnect to screen
[...]
template1=> \connect template1
pg_krb5_init: krb5_cc_get_principal: No credentials cache found
----------------------

There's not really a solution to this issue (besides exit psql, reset
the environment variable, and restart psql) unless we allow the ccache
to be set in psql and then be propogated down to libpq.  I actually
hacked up some .profile/.logout scripts to avoid this problem by forcing
the cache to a fixed known-spot under my home directory.  I don't think
that really works for this though because everything is running as one
uid under Apache and having a fixed long-term filename would present
some additional security issues.

> It sure seems like somebody is solving the wrong problem, and doing it
> with a sledgehammer...

The sledgehammer approach would be to require people to run everything
as CGIs.  This does work, of course, but is rather painful on a number
of levels...

    Thanks,

        Stephen