Thread: Re: GSS Auth issue when user member of lots of AD groups

[ pgsql-committers is completely inappropriate, redirecting to -bugs ]

Chris Gooch <cgooch@bamfunds.com> writes:
> GSS authentication is working for users with small number of AD
> groups but getting below error when a user has larger number of
> groups. I believe it might to token size related, but they don't
> have issues when authenticating with Kerberos/GSS to other
> applications, only with Postgres.

> failed: GSSAPI context establishment error: The routine must be called again to complete its function: Unknown error

Hmm.  That must be coming from this bit in libpq:

    /* Must have output.length > 0 */
    if (output.length > PQ_GSS_SEND_BUFFER_SIZE - sizeof(uint32))
    {
        pg_GSS_error(libpq_gettext("GSSAPI context establishment error"),
                     conn, major, minor);
        gss_release_buffer(&minor, &output);
        return PGRES_POLLING_FAILED;
    }

which makes it look like gss_init_sec_context wants us to send a
packet larger than PQ_GSS_SEND_BUFFER_SIZE, which perhaps is a
plausible thing to happen if the user belongs to enough groups.

Unfortunately, elsewhere in the same file:

 * NOTE: The client and server have to agree on the max packet size,
 * because we have to pass an entire packet to GSSAPI at a time and we
 * don't want the other side to send arbitrarily huge packets as we
 * would have to allocate memory for them to then pass them to GSSAPI.
 *
 * Therefore, these two #define's are effectively part of the protocol
 * spec and can't ever be changed.
 */
#define PQ_GSS_SEND_BUFFER_SIZE 16384
#define PQ_GSS_RECV_BUFFER_SIZE 16384

Not sure where to go from here.  Unfortunately the person who
was mostly responsible for this code has left the project...

            regards, tom lane

On Thu, May 22, 2025 at 9:57 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I'm wondering though if this isn't just pushing the problem out a
> little further.  Is there a good reason to think 64K is enough?

Microsoft docs [1] seem to imply that there are still a bunch of
existing problems if you try to go much higher, though it is possible
to do so with registry tweaks. Looks like they default to 48k.

Maybe we should consider making the max incoming ticket size
configurable, so users that really need a bigger one can deal with the
DoS risk without it affecting everyone else. (A limit on outgoing
tickets probably doesn't make too much sense; I imagine you're going
to use the ticket that GSSAPI hands you, no matter how big it is,
because it's not as if you have a choice.)

--Jacob

[1]
https://learn.microsoft.com/en-us/troubleshoot/windows-server/windows-security/kerberos-authentication-problems-if-user-belongs-to-groups#known-issues-that-affect-maxtokensize

Jacob Champion <jacob.champion@enterprisedb.com> writes:
> On Thu, May 22, 2025 at 9:57 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I'm wondering though if this isn't just pushing the problem out a
>> little further.  Is there a good reason to think 64K is enough?

> Microsoft docs [1] seem to imply that there are still a bunch of
> existing problems if you try to go much higher, though it is possible
> to do so with registry tweaks. Looks like they default to 48k.

> Maybe we should consider making the max incoming ticket size
> configurable, so users that really need a bigger one can deal with the
> DoS risk without it affecting everyone else. (A limit on outgoing
> tickets probably doesn't make too much sense; I imagine you're going
> to use the ticket that GSSAPI hands you, no matter how big it is,
> because it's not as if you have a choice.)

Yeah, but we don't want to change the packet size used after the
initial exchange, because that would create compatibility issues
in cases that aren't failing today.  I didn't look at the code
to see if we can easily use a different buffer size during
the authentication exchange.  If we can, I'd be inclined to goose
it up to 128K or so.  Given Chris' point that should be plenty,
so I don't feel a need to expose a knob.

            regards, tom lane

Chris Gooch <cgooch@bamfunds.com> writes:
> Hi Both, Will anyone be working on a fix for this bug?

Yeah, I will pick it up (unless Jacob's already on it?)

I believe we've agreed that it'd be sufficient if we allow the
packets exchanged during the auth phase to be up to 64K or so,
but once we reach the point where we're able to split the
data on arbitrary boundaries, keep the packet size at 16K
for cross-version compatibility.  I did look at the code
briefly, and this seems like it'd require releasing and
re-alloc'ing the buffers, but that's surely doable.

            regards, tom lane

On Thu, May 22, 2025 at 05:04:32PM +0000, Chris Gooch wrote:
> It now makes sense to me. I believe the KDC will not allow tokens
> larger than 65535 bytes, so feel it is safe from a GSS perspective.

The KDC protocol over TCP uses 32-bit unsigned PDU lengths in network
byte order, of which the high bit is reserved and must be zero.  ASN.1
supports much larger lengths still.  The protocol easily supports very
large tickets, therefore very large initial security context tokens.

The architecture of having the user's SIDs be included in the user's
service tickets was very useful as an optimization for a long time, but
as the number of SIDs increases this optimization becomes more of an
albatross.

Nico
-- 

Chris Gooch <cgooch@bamfunds.com> writes:
> Thanks Tom. More than happy to give these patches a test as I can easily reproduce the bug.

Great!

> Are there specific test cases you would like me to try?

Obviously, check if it works for the user with lots of AD groups.
But beyond that, just check it in everyday use.  All I've done is
to run our "kerberos" regression test, and I don't have a huge
amount of faith in the coverage of that.

            regards, tom lane

I wrote:
> Chris Gooch <cgooch@bamfunds.com> writes:
>> Are there specific test cases you would like me to try?

> Obviously, check if it works for the user with lots of AD groups.
> But beyond that, just check it in everyday use.

Oh --- one specific scenario that would be good to check is to
verify that patched libpq can still use GSS with unpatched
server and vice versa, so long as the ticket doesn't exceed 16K.

            regards, tom lane

Thanks again for sharing the patches Tom.

I have just tested the following combinations on v17 and all working as expected, both with tickets less thank 16k and
greaterthan 16k. 

1. Patched Client to Unpatched Server
2. Unpatched Client to Patched Server
3. Patched Client to Patched Server


Thanks,
Chris

-----Original Message-----
From: Tom Lane <tgl@sss.pgh.pa.us>
Sent: Sunday, May 25, 2025 4:51 PM
To: Chris Gooch <cgooch@bamfunds.com>
Cc: Jacob Champion <jacob.champion@enterprisedb.com>; pgsql-bugs@lists.postgresql.org
Subject: Re: [EXT] Re: GSS Auth issue when user member of lots of AD groups

I wrote:
> Chris Gooch <cgooch@bamfunds.com> writes:
>> Are there specific test cases you would like me to try?

> Obviously, check if it works for the user with lots of AD groups.
> But beyond that, just check it in everyday use.

Oh --- one specific scenario that would be good to check is to verify that patched libpq can still use GSS with
unpatchedserver and vice versa, so long as the ticket doesn't exceed 16K. 

            regards, tom lane

This email and any attachments should not be construed as an offer or recommendation to sell or buy or a solicitation
ofan offer to sell or buy any specific security, fund or instrument or to participate in any particular investment
strategy.The information contained herein is given as of a certain date and does not purport to give information as of
anyother date. Although the information presented herein has been obtained from sources we believe to be reliable, no
representationor warranty, expressed or implied, is made as to the accuracy or completeness of that information. Past
performanceis not indicative of future results. 

CONFIDENTIALITY NOTICE: This message and any attachment are confidential. If you are not the intended recipient, please
telephoneor email the sender and delete this message and any attachment from your system. If you are not the intended
recipientyou must not copy this message or attachment or disclose the contents to any other persons. 

Balyasny Asset Management (UK) LLP is authorised and regulated by the Financial Conduct Authority in the UK. Balyasny
AssetManagement LP is registered as an Investment Advisor with the Securities and Exchange Commission in the USA. 

BAM prohibits all personnel from having any business related communications over text message or other unapproved
communicationapplications. Unless pre-approved, BAM employees are only permitted to communicate over email, Bloomberg
andBAM telephone lines.