Thread: Correct documentation for protocol version

On 2025/04/10 18:52, Dave Cramer wrote:
> Greetings,
> 
> The current docs say that if a client asks for a protocol that the backend doesn't support, it will return the
newest minorversion.
https://www.postgresql.org/docs/current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-NEGOTIATEPROTOCOLVERSION
<https://www.postgresql.org/docs/current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-NEGOTIATEPROTOCOLVERSION>
> 
> However that isn't what it returns. It actually returns the entire newest protocol that it supports. Attached is a
patchto fix the docs.
 

As far as I read the code, the server returns the protocol version requested by
the client if it's less than or equal to the latest version the server supports.
Otherwise, it returns the latest supported version. So the proposed description
doesn't seem accurate either, does it?

+         Major is in the upper 16 bits and the lower in the low 16 bits.

To match the style of similar descriptions, how about rephrasing it as:
"The most significant 16 bits are the major version number, and the least
significant 16 bits are the minor version number”?

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

On Thu, Apr 10, 2025 at 7:41 AM Dave Cramer <davecramer@gmail.com> wrote:
> Done in new patch attached

I think this patch splices a sentence:

> +         not equal to the version the server supports.
>           message.

+1 for clarifying the message description; it has vaguely bothered me
for a while [1]. :)

--Jacob

[1] https://www.postgresql.org/message-id/7a43f140-60e7-87ce-cf68-ea6d40e4b904%40timescale.com

On 2025/04/10 23:40, Dave Cramer wrote:
> 
> On Thu, 10 Apr 2025 at 09:54, Fujii Masao <masao.fujii@oss.nttdata.com <mailto:masao.fujii@oss.nttdata.com>> wrote:
> 
> 
> 
>     On 2025/04/10 18:52, Dave Cramer wrote:
>      > Greetings,
>      >
>      > The current docs say that if a client asks for a protocol that the backend doesn't support, it will return the
newest minorversion.
https://www.postgresql.org/docs/current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-NEGOTIATEPROTOCOLVERSION
<https://www.postgresql.org/docs/current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-NEGOTIATEPROTOCOLVERSION>
<https://www.postgresql.org/docs/current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-NEGOTIATEPROTOCOLVERSION
<https://www.postgresql.org/docs/current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-NEGOTIATEPROTOCOLVERSION>>
>      >
>      > However that isn't what it returns. It actually returns the entire newest protocol that it supports. Attached
isa patch to fix the docs.
 
> 
>     As far as I read the code, the server returns the protocol version requested by
>     the client if it's less than or equal to the latest version the server supports.
>     Otherwise, it returns the latest supported version. So the proposed description
>     doesn't seem accurate either, does it?
> 
> I've added a note as to when this is sent. AFAICT through testing and the fact that the pgjdbc driver has never
handledthis message
 
> I'm pretty sure it only sends this message if the requested protocol is not equal to the protocol the server
supports.

No, the message is also sent when the client requests protocol options that
the server doesn't recognize. In that case, if the client requests an older
protocol version along with unknown options, the server responds with
the requested protocol version, not the latest one.


>     +         Major is in the upper 16 bits and the lower in the low 16 bits.
> 
>     To match the style of similar descriptions, how about rephrasing it as:
>     "The most significant 16 bits are the major version number, and the least
>     significant 16 bits are the minor version number”?
> 
> Done in new patch attached

Thanks!

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

On 2025/04/11 0:49, Dave Cramer wrote:
> 
> 
> On Thu, 10 Apr 2025 at 11:17, Fujii Masao <masao.fujii@oss.nttdata.com <mailto:masao.fujii@oss.nttdata.com>> wrote:
> 
> 
> 
>     On 2025/04/10 23:40, Dave Cramer wrote:
>      >
>      > On Thu, 10 Apr 2025 at 09:54, Fujii Masao <masao.fujii@oss.nttdata.com <mailto:masao.fujii@oss.nttdata.com>
<mailto:masao.fujii@oss.nttdata.com<mailto:masao.fujii@oss.nttdata.com>>> wrote:
 
>      >
>      >
>      >
>      >     On 2025/04/10 18:52, Dave Cramer wrote:
>      >      > Greetings,
>      >      >
>      >      > The current docs say that if a client asks for a protocol that the backend doesn't support, it will
returnthe newest minor version.
https://www.postgresql.org/docs/current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-NEGOTIATEPROTOCOLVERSION
<https://www.postgresql.org/docs/current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-NEGOTIATEPROTOCOLVERSION>
<https://www.postgresql.org/docs/current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-NEGOTIATEPROTOCOLVERSION
<https://www.postgresql.org/docs/current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-NEGOTIATEPROTOCOLVERSION>>
<https://www.postgresql.org/docs/current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-NEGOTIATEPROTOCOLVERSION
<https://www.postgresql.org/docs/current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-NEGOTIATEPROTOCOLVERSION>
<https://www.postgresql.org/docs/current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-
>     NEGOTIATEPROTOCOLVERSION
<https://www.postgresql.org/docs/current/protocol-message-formats.html#PROTOCOL-MESSAGE-FORMATS-NEGOTIATEPROTOCOLVERSION>>>
>      >      >
>      >      > However that isn't what it returns. It actually returns the entire newest protocol that it supports.
Attachedis a patch to fix the docs.
 
>      >
>      >     As far as I read the code, the server returns the protocol version requested by
>      >     the client if it's less than or equal to the latest version the server supports.
>      >     Otherwise, it returns the latest supported version. So the proposed description
>      >     doesn't seem accurate either, does it?
>      >
>      > I've added a note as to when this is sent. AFAICT through testing and the fact that the pgjdbc driver has
neverhandled this message
 
>      > I'm pretty sure it only sends this message if the requested protocol is not equal to the protocol the server
supports.
> 
>     No, the message is also sent when the client requests protocol options that
>     the server doesn't recognize. In that case, if the client requests an older
>     protocol version along with unknown options, the server responds with
>     the requested protocol version, not the latest one.
> 
> 
> OK, I hadn't contemplated the unrecognized options. However AFAICT the documentation in the code state
> 
> /*
> * If the client requested a newer protocol version or if the client
> * requested any protocol options we didn't recognize, let them know
> * the newest minor protocol version we do support and the names of
> * any unrecognized options.
> */
> 
> and the code agrees that we send the latest protocol version that we support, not the requested one.

SendNegotiateProtocolVersion() sends the NegotiateProtocolVersion message and
returns the protocol version stored in the FrontendProtocol variable:

    static void
    SendNegotiateProtocolVersion(List *unrecognized_protocol_options)
    {
        StringInfoData buf;
        ListCell   *lc;
        
        pq_beginmessage(&buf, PqMsg_NegotiateProtocolVersion);
        pq_sendint32(&buf, FrontendProtocol);

FrontendProtocol is set by ProcessStartupPacket() as Min(proto, PG_PROTOCOL_LATEST),
where proto is the protocol version requested by the client. So, if the client
requests an older version, i.e., proto is smaller than PG_PROTOCOL_LATEST,
the server may respond with that requested version:

    /*
     * Set FrontendProtocol now so that ereport() knows what format to send if
     * we fail during startup. We use the protocol version requested by the
     * client unless it's higher than the latest version we support. It's
     * possible that error message fields might look different in newer
     * protocol versions, but that's something those new clients should be
     * able to deal with.
     */
    FrontendProtocol = Min(proto, PG_PROTOCOL_LATEST);

Am I missing something?

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

On 2025/04/11 5:17, Dave Cramer wrote:
> No, you are correct.
> 
> See new patch

Thanks for updating the patch!

-         Identifies the message as a protocol version negotiation
+         Identifies the message as a protocol version negotiation.
+         The server sends this message if the requested protocol is
+         not equal to the version the server supports or the client
+         requests protocol options that are not recognized.
           message.

You added the sentence starting with "The server sends..."
between "negotiation" and "message", but it should be placed
after "message", right?

Even though the requested version is not equal to the latest
version that the server supports, if it's older than
the latest one, the message is not sent. So how about
wording it like this instead:

-------------
Identifies the message as a protocol version negotiation message.
The server sends this message when the client requests a newer
protocol version than the server supports, or when the client
includes protocol options that the server does not recognize.
-------------

+         The protcol version requested by the client unless it is higher than the
+         latest version we support in which case the latest protocol version we support.

Maybe rewording this for clarity and using “the server
instead of “we” would help. For example:

-------------
The latest protocol version supported by the server if the client
requests a newer protocol version than the server supports.
The protocol version requested by the client, otherwise.
-------------


Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

On 2025/04/11 18:27, Dave Cramer wrote:
> 
> 
> On Fri, 11 Apr 2025 at 05:05, Fujii Masao <masao.fujii@oss.nttdata.com <mailto:masao.fujii@oss.nttdata.com>> wrote:
> 
> 
> 
>     On 2025/04/11 5:17, Dave Cramer wrote:
>      > No, you are correct.
>      >
>      > See new patch
> 
>     Thanks for updating the patch!
> 
>     -         Identifies the message as a protocol version negotiation
>     +         Identifies the message as a protocol version negotiation.
>     +         The server sends this message if the requested protocol is
>     +         not equal to the version the server supports or the client
>     +         requests protocol options that are not recognized.
>                 message.
> 
>     You added the sentence starting with "The server sends..."
>     between "negotiation" and "message", but it should be placed
>     after "message", right?
> 
>     Even though the requested version is not equal to the latest
>     version that the server supports, if it's older than
>     the latest one, the message is not sent. So how about
>     wording it like this instead:
> 
>     -------------
>     Identifies the message as a protocol version negotiation message.
>     The server sends this message when the client requests a newer
>     protocol version than the server supports, or when the client
>     includes protocol options that the server does not recognize.
>     -------------
> 
>     +         The protcol version requested by the client unless it is higher than the
>     +         latest version we support in which case the latest protocol version we support.
> 
>     Maybe rewording this for clarity and using “the server
>     instead of “we” would help. For example:
> 
>     -------------
>     The latest protocol version supported by the server if the client
>     requests a newer protocol version than the server supports.
>     The protocol version requested by the client, otherwise.
>     -------------
> 
> 
> Reworded as suggested

Thanks for updating the patch!


While checking the code in older branches, I noticed that the returned
protocol version is always the latest version supported by the server.
However, as we discussed, in master, the server may return the version
requested by the client. The change was introduced in commit 516b87502dc.
So, probably we'll need to update the documentation differently for
master and the older branches.


The patch adds a new explanation about when the NegotiateProtocolVersion
message is sent. But a similar explanation already exists in protocol.sgml:

       <term>NegotiateProtocolVersion</term>
       <listitem>
        <para>
         The server does not support the minor protocol version requested
         by the client, but does support an earlier version of the protocol;
         this message indicates the highest supported minor version.  This
         message will also be sent if the client requested unsupported protocol
         options (i.e., beginning with <literal>_pq_.</literal>) in the
         startup packet.

Given that, I'm now wondering if the new description in the patch
might be redundant.


Also, your original concern was that the phrase "Newest minor protocol version"
is inaccurate since the field contains both major and minor version numbers
(e.g., 3.2). However, based on other usage in protocol.sgml and source
comments in related code, "minor version" seems to refer to the full version
like 3.2, i.e., not just the minor part, so we might not need to reword it
after all.

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

On Fri, 11 Apr 2025 at 21:39, Fujii Masao <masao.fujii@oss.nttdata.com> wrote:
> While checking the code in older branches, I noticed that the returned
> protocol version is always the latest version supported by the server.
> However, as we discussed, in master, the server may return the version
> requested by the client. The change was introduced in commit 516b87502dc.
> So, probably we'll need to update the documentation differently for
> master and the older branches.

No need for different docs. Given that older branches only support 3.0
protocol, there's no way for a client to request a version earlier
than the "latest version supported by the server".

> The patch adds a new explanation about when the NegotiateProtocolVersion
> message is sent. But a similar explanation already exists in protocol.sgml:

Side-comment: I think our protocol docs are pretty annoyingly spread
across two pages.

> Given that, I'm now wondering if the new description in the patch
> might be redundant.
>
>
> Also, your original concern was that the phrase "Newest minor protocol version"
> is inaccurate since the field contains both major and minor version numbers
> (e.g., 3.2). However, based on other usage in protocol.sgml and source
> comments in related code, "minor version" seems to refer to the full version
> like 3.2, i.e., not just the minor part, so we might not need to reword it
> after all.

I quite like the new wording from Dave so +1 from me. I also think for
protocol docs it's especially important to be very precise and leave
very little room for interpretation.

One thing that we should probably clarify though (which was somewhat
clarified in the previous wording) is that we only send this message
if the client requested a major version that the major version that
the server supports. i.e. we will never send a
NegotiateProtocolVersion message to 3.2 if the client requested 4.0.

On Fri, 11 Apr 2025 at 22:57, Dave Cramer <davecramer@gmail.com> wrote:
> Well this isn't quite true since if you request 3.0 and have invalid options it will return 3.0, which is not the
highestsupported minor version.
 

Probably good to update this section too then to be similarly correct
as your already updated section. Maybe also good to clarify further
that the version that the server responds with is the protocol version
that will be used during the following communication.