Thread: Doing authentication in backend
If we did this the straightforward way (exchange authentication packets after fork()) then rogue clients could connect, start a backend, twiddle thumbs, never finish the authentication exchange, meanwhile having filled up the limit on the number of connections. Somehow the backends would have to report back to the postmaster that the authentication passed. But then an attacker could easily fill up the system's process table with this approach. If you in turn put a cap on that to save your system at large, you're back to having DoS'ed your database server. Then you would have to put a timeout on the completion of the authentication sequence. This would be a fairly tricky thing to configure given the various choices of ways to authenticate, including interactive ones. ISTM that there is some merit in having authentication happen *before* doing much else, especially allocating per-connection resources. Comments? -- Peter Eisentraut peter_e@gmx.net http://funkturm.homeip.net/~peter
Peter Eisentraut <peter_e@gmx.net> writes:
> If we did this the straightforward way (exchange authentication packets
> after fork()) then rogue clients could connect, start a backend, twiddle
> thumbs, never finish the authentication exchange, meanwhile having filled
> up the limit on the number of connections.
True, but don't fool yourself that a similar DOS attack is not possible
now. The resource limit that an attacker can hit now is the maximum
number of open file descriptors for the single postmaster process, which
may be quite a lot lower than the maximum number of process table
entries, depending on how your system is configured.
Also note that we could easily fix things so that the max-number-of-
backends limit is not checked until we have passed the authentication
procedure. A PM child that's still busy authenticating doesn't have
to count.
> ISTM that there is some merit in having authentication happen *before*
> doing much else, especially allocating per-connection resources.
Sure, which is why the postmaster is written the way it is. But you
have to be willing to code the postmaster in a way that prevents it from
blocking on behalf of one client. We don't have that now for IDENT,
we are about to not have it for PAM, and I don't see a lot of enthusiasm
out there for adhering to those coding rules with the rigidity needed to
realize the theoretical benefit.
Another problem with the present setup is total cost of servicing each
connection request. We've seen several complaints about connection-
refused problems under heavy load, occurring because the single
postmaster process simply can't service the requests quickly enough to
keep its accept() queue from overflowing.
Forking the postmaster (without an exec) is a relatively cheap
operation, since the PM has only a small amount of writable data and
very few open files. I believe forking before authenticating would
improve the accept-queue-overflow problem by reducing the amount of
work done before the PM can accept() another connection request.
Moreover, if we went over to fork-before-authenticate, we could rip out
all the poor man's multitasking code that's in the postmaster. That
would make the PM simpler, more understandable, and ultimately more
reliable.
So on the whole I think changing would be a win.
regards, tom lane
Tom Lane writes: > Also note that we could easily fix things so that the max-number-of- > backends limit is not checked until we have passed the authentication > procedure. A PM child that's still busy authenticating doesn't have > to count. How does the postmaster know? And if the postmaster does get to know, what does it do with children it has "accidentally" allowed in excess? -- Peter Eisentraut peter_e@gmx.net http://funkturm.homeip.net/~peter
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> Also note that we could easily fix things so that the max-number-of-
>> backends limit is not checked until we have passed the authentication
>> procedure. A PM child that's still busy authenticating doesn't have
>> to count.
> How does the postmaster know? And if the postmaster does get to know,
> what does it do with children it has "accidentally" allowed in excess?
The postmaster doesn't have to know, nor should it be in the business of
enforcing the MaxBackends limit. It should just spawn off children
(though maybe we should put a limit on total children, somewhat higher
than MaxBackends, as a crude form of preventing runaway resource usage
under a DOS attack).
A spawned child will first proceed with the authorization cycle; if it
fails, it just exits. If it succeeds, it will then try to become a
backend. When it tries to insert an entry into the PROC array, if
there's not an available slot then you lose (send "too many clients"
failure message to client, and exit). We might have to reorganize the
code a little bit so that this exit happens cleanly before anything else
is done to shared memory, but it's surely doable.
The PM itself has no real need to distinguish children that are active
backends from those that are still doing authentication, AFAICS.
regards, tom lane
On Thu, Jun 14, 2001 at 01:42:26PM -0400, Tom Lane wrote: > Also note that we could easily fix things so that the max-number-of- > backends limit is not checked until we have passed the authentication > procedure. A PM child that's still busy authenticating doesn't have > to count. And impose a very short timeout on authentication. > Another problem with the present setup is total cost of servicing each > connection request. We've seen several complaints about connection- > refused problems under heavy load, occurring because the single > postmaster process simply can't service the requests quickly enough to > keep its accept() queue from overflowing. This last could also be addressed (along with Solaris's Unix Sockets problem!) by changing the second argument to listen(2) from the current SOMAXCONN -- which is 5 in Solaris 2.7 -- to 127. See the six-page discussion in Stevens UNPv1 beginning at page 93. This is not to say we shouldn't fork before authentication, for the above and other reasons, but the fix to listen(2)'s argument should happen anyway. Nathan Myers ncm@zembu.com
ncm@zembu.com (Nathan Myers) writes:
> On Thu, Jun 14, 2001 at 01:42:26PM -0400, Tom Lane wrote:
>> Also note that we could easily fix things so that the max-number-of-
>> backends limit is not checked until we have passed the authentication
>> procedure. A PM child that's still busy authenticating doesn't have
>> to count.
> And impose a very short timeout on authentication.
Yes. There's no time limit at present, but it will be easy to add one
after we change to fork-before-authenticate (since each PM child can
have its own itimer).
> This last could also be addressed (along with Solaris's Unix Sockets
> problem!) by changing the second argument to listen(2) from the current
> SOMAXCONN -- which is 5 in Solaris 2.7 -- to 127. See the six-page
> discussion in Stevens UNPv1 beginning at page 93.
Unfortunately I only have Stevens' first edition, and it doesn't seem
to have any such advice in it. Why is it a good idea to ignore the
platform's specification of SOMAXCONN? Seems like on non-broken
platforms, that would do more harm than good.
regards, tom lane
On Sat, Jun 16, 2001 at 01:02:15PM -0400, Tom Lane wrote: > ncm@zembu.com (Nathan Myers) writes: > > On Thu, Jun 14, 2001 at 01:42:26PM -0400, Tom Lane wrote: > > This last could also be addressed (along with Solaris's Unix Sockets > > problem!) by changing the second argument to listen(2) from the current > > SOMAXCONN -- which is 5 in Solaris 2.7 -- to 127. See the six-page > > discussion in Stevens UNPv1 beginning at page 93. > > Unfortunately I only have Stevens' first edition, and it doesn't seem > to have any such advice in it. Why is it a good idea to ignore the > platform's specification of SOMAXCONN? Seems like on non-broken > platforms, that would do more harm than good. The second edition is easily worth the price, for any number of reasons. Do you want me to type in all six pages? (I'll fax you a copy if you ask.) He includes graphs of sample daemon transaction rates for different settings of that argument, from SOMAXCONN on up, as well as analyses of what is going on, including diagrams. The short description is that half-completed connections occupy a sort of foyer or vestibule (my terms). The second argument to listen(2), usually called "backlog", is defined vaguely in Posix and therefore has various meanings on different systems, but is supposed to limit how many half-open connections are allowed to wait there. I don't have the book at home, but IIRC, Solarix's interpretation is unusually strict, which causes it to reject connections much more aggressively for a given value. Independently of that, the low value that was originally suggested just turned out to be a bad guess. The mistake got worse as longer, fatter pipes got deployed and protocols that did more opens got popular. Posix says that a backlog value more than the platform's maximum gets folded, so there is no danger in exceeding it. On Solaris 2.7, SOMAXCONN is still 5 in the headers, but larger values are both legal and effective. A parameter might reasonably be added to the config file to crank the backlog value down on installations where kernel memory is scarce and other daemons must compete with PG for connection resources. I have seen reports that it is this strict interpretation, and the too-low backlog passed to listen(2), that accounts for reports of Unix sockets working poorly on Solaris. (Certainly it is hard to believe that Sun could not make their Unix sockets work right!) Nathan Myers ncm@zembu.com