Thread: real/float example for testlibpq3
Hi everyone, Would adding additional examples to testlibpq3.c on handling more data types be helpful? I've attached a patch adding how to handle a REAL to current example. Regards, Mark
Attachment
Hi Mark, Fetching a "real" type field from libpq doesn't look that intuitive. So this example is super helpful. Thanks. I am wondering whether we should provide PQgetfloatvalue() which does what that example shows but transparently. It will return a float value of the given field instead of char *. That will standardize the way to fetch real typed values in libpq. That leads to the next question. Do we need to introduce different PQget*value() for standard C/SQL data types. -- Best Wishes, Ashutosh Bapat On Fri, Feb 25, 2022 at 3:12 AM Mark Wong <markwkm@gmail.com> wrote: > > Hi everyone, > > Would adding additional examples to testlibpq3.c on handling more data > types be helpful? I've attached a patch adding how to handle a REAL to > current example. > > Regards, > Mark
Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> writes:
> I am wondering whether we should provide PQgetfloatvalue() which does
> what that example shows but transparently. It will return a float
> value of the given field instead of char *.
I'm against that, precisely because ...
> That will standardize the
> way to fetch real typed values in libpq. That leads to the next
> question. Do we need to introduce different PQget*value() for standard
> C/SQL data types.
... I do not want to go here. Where would you stop? How would you
deal with cross-machine inconsistencies in integer widths?
regards, tom lane
On 02/28/22 10:19, Tom Lane wrote: >> That will standardize the >> way to fetch real typed values in libpq. That leads to the next >> question. Do we need to introduce different PQget*value() for standard >> C/SQL data types. > > ... I do not want to go here. Where would you stop? How would you > deal with cross-machine inconsistencies in integer widths? This stimulates a question for me. Not just libpq, but all drivers implemented in whatever language, if they wish to support binary protocol, depend on knowing what the committed send/recv wire formats are for whichever types they mean to support. In the current state of affairs, what's considered the ur-source of that information? I have often seen those formats documented in code comments, usually above the recv function in the .c file for a given adt. Have we got any more, well, machine-readable collection of that knowledge? Regards, -Chap
Chapman Flack <chap@anastigmatix.net> writes:
> This stimulates a question for me.
> Not just libpq, but all drivers implemented in whatever language, if they
> wish to support binary protocol, depend on knowing what the committed
> send/recv wire formats are for whichever types they mean to support.
> In the current state of affairs, what's considered the ur-source of that
> information?
The source code for the type's send/receive functions :-(. One could
wish for something better, but no one has stepped up to produce such
documentation.
regards, tom lane
On Mon, 28 Feb 2022 at 17:50, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Chapman Flack <chap@anastigmatix.net> writes: > > In the current state of affairs, what's considered the ur-source of that > > information? > > The source code for the type's send/receive functions :-(. One could > wish for something better, but no one has stepped up to produce such > documentation. Fwiw the client library I heard of attempting to have good binary mode support was the Crystal language client https://github.com/will/crystal-pg. I think he was aiming for full coverage of the built-in data types. That might make a good reference implementation to write up documentation from. He probably uncovered some corner cases in development that one might not find from just inspection of the server code. -- greg
On Wed, Mar 30, 2022 at 01:16:37PM -0400, Greg Stark wrote: > On Mon, 28 Feb 2022 at 17:50, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > > > Chapman Flack <chap@anastigmatix.net> writes: > > > In the current state of affairs, what's considered the ur-source of that > > > information? > > > > The source code for the type's send/receive functions :-(. One could > > wish for something better, but no one has stepped up to produce such > > documentation. > > Fwiw the client library I heard of attempting to have good binary mode > support was the Crystal language client > https://github.com/will/crystal-pg. I think he was aiming for full > coverage of the built-in data types. That might make a good reference > implementation to write up documentation from. He probably uncovered > some corner cases in development that one might not find from just > inspection of the server code. Checking in for quick feedback to see if this refactor makes sense. I've created a function for each data type with the idea that an example for handling a specific data type can be more easily reviewed by looking in a single place. I've added examples for REAL, TIMESTAMP WITHOUT TIME ZONE, and BOOLEAN to try to illustrate how testlibpq3.sql and testlibpq3.c will grow if this is a good way to go. Regards, Mark
Attachment
On Tue, Jun 14, 2022 at 1:40 PM Mark Wong <markwkm@gmail.com> wrote:
> Checking in for quick feedback to see if this refactor makes sense.
>
> I've created a function for each data type with the idea that an example
> for handling a specific data type can be more easily reviewed by looking
> in a single place.
>
> I've added examples for REAL, TIMESTAMP WITHOUT TIME ZONE, and BOOLEAN
> to try to illustrate how testlibpq3.sql and testlibpq3.c will grow if
> this is a good way to go.
I think this could be a reasonable kind of thing to do, but I think
you left the "ts" output out of the example output in the comments,
and also your code's apparently not portable, because my compiler is
OK with testlibpq3 right now, but with your patch it emits lengthy
unhappy moaning, starting with:
testlibpq3.c:88:10: error: expected ')'
uint64_t ntohll(uint64_t);
^
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_endian.h:140:25:
note: expanded from macro 'ntohll'
#define ntohll(x) __DARWIN_OSSwapInt64(x)
^
Apparently macOS defines ntohll as a macro, and it's not amused by
your attempt to include a function prototype for it.
I'm not sure that we want to let these test programs grow very large.
In particular, I don't think we should let ourselves get sucked into
adding an example for every data type under the sun -- if that's
wanted, the solution is perhaps to add documentation for the binary
formats, not hide impromptu documentation inside a test program. But
doing this much seems OK to me.
--
Robert Haas
EDB: http://www.enterprisedb.com
Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Jun 14, 2022 at 1:40 PM Mark Wong <markwkm@gmail.com> wrote:
>> I've created a function for each data type with the idea that an example
>> for handling a specific data type can be more easily reviewed by looking
>> in a single place.
>> I've added examples for REAL, TIMESTAMP WITHOUT TIME ZONE, and BOOLEAN
>> to try to illustrate how testlibpq3.sql and testlibpq3.c will grow if
>> this is a good way to go.
> I'm not sure that we want to let these test programs grow very large.
> In particular, I don't think we should let ourselves get sucked into
> adding an example for every data type under the sun -- if that's
> wanted, the solution is perhaps to add documentation for the binary
> formats, not hide impromptu documentation inside a test program. But
> doing this much seems OK to me.
Yeah, "hiding impromptu documentation inside a test program" is what
this looks like, and I'm not sure that's a reasonable way to go.
(1) Who's going to think to look in src/test/examples/testlibpq3.c for
documentation of binary formats?
(2) The useful details are likely to get buried in notational and
portability concerns, as I think your build failure illustrates.
(3) I bet few if any packagers install these files, so that the new
info would be unavailable to many people.
I think some new appendix in the main SGML docs would be the appropriate
place if we want to provide real documentation.
regards, tom lane
On Thu, Jun 16, 2022 at 03:41:50PM -0400, Tom Lane wrote: > Robert Haas <robertmhaas@gmail.com> writes: > > On Tue, Jun 14, 2022 at 1:40 PM Mark Wong <markwkm@gmail.com> wrote: > >> I've created a function for each data type with the idea that an example > >> for handling a specific data type can be more easily reviewed by looking > >> in a single place. > >> I've added examples for REAL, TIMESTAMP WITHOUT TIME ZONE, and BOOLEAN > >> to try to illustrate how testlibpq3.sql and testlibpq3.c will grow if > >> this is a good way to go. > > > I'm not sure that we want to let these test programs grow very large. > > In particular, I don't think we should let ourselves get sucked into > > adding an example for every data type under the sun -- if that's > > wanted, the solution is perhaps to add documentation for the binary > > formats, not hide impromptu documentation inside a test program. But > > doing this much seems OK to me. > > Yeah, "hiding impromptu documentation inside a test program" is what > this looks like, and I'm not sure that's a reasonable way to go. > > (1) Who's going to think to look in src/test/examples/testlibpq3.c for > documentation of binary formats? > > (2) The useful details are likely to get buried in notational and > portability concerns, as I think your build failure illustrates. > > (3) I bet few if any packagers install these files, so that the new > info would be unavailable to many people. > > I think some new appendix in the main SGML docs would be the appropriate > place if we want to provide real documentation. Ok, I'll leave the testlibpq3.c as it was then. If it's worth keeping any of those changes, then I can remove the timestamp example because of the ntohll() portability since that is trivial. I'll start a new appendix and share again when I have something to show. Regards, Mark -- Mark Wong EDB: http://www.enterprisedb.com
On Thu, Jun 16, 2022 at 03:41:50PM -0400, Tom Lane wrote: > Robert Haas <robertmhaas@gmail.com> writes: > > On Tue, Jun 14, 2022 at 1:40 PM Mark Wong <markwkm@gmail.com> wrote: > >> I've created a function for each data type with the idea that an example > >> for handling a specific data type can be more easily reviewed by looking > >> in a single place. > >> I've added examples for REAL, TIMESTAMP WITHOUT TIME ZONE, and BOOLEAN > >> to try to illustrate how testlibpq3.sql and testlibpq3.c will grow if > >> this is a good way to go. > > > I'm not sure that we want to let these test programs grow very large. > > In particular, I don't think we should let ourselves get sucked into > > adding an example for every data type under the sun -- if that's > > wanted, the solution is perhaps to add documentation for the binary > > formats, not hide impromptu documentation inside a test program. But > > doing this much seems OK to me. > > Yeah, "hiding impromptu documentation inside a test program" is what > this looks like, and I'm not sure that's a reasonable way to go. > > (1) Who's going to think to look in src/test/examples/testlibpq3.c for > documentation of binary formats? > > (2) The useful details are likely to get buried in notational and > portability concerns, as I think your build failure illustrates. > > (3) I bet few if any packagers install these files, so that the new > info would be unavailable to many people. > > I think some new appendix in the main SGML docs would be the appropriate > place if we want to provide real documentation. Just wanted to do another quick check in before I go too far. How do does this start look? Extending the libpq section with a code snippet and description per data type? Regards, Mark
Attachment
On 01.07.22 19:18, Mark Wong wrote: > Just wanted to do another quick check in before I go too far. How do > does this start look? Extending the libpq section with a code snippet > and description per data type? If we want to start documenting the binary format for all data types, it should go into the "Data Types" chapter. There, we already document the text format in great detail, so putting the binary format in there as well would make sense. The libpq chapter could then contain some examples of applying that information in the context of libpq. But the libpq documentation shouldn't be the primary place where the formats are documented.
Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
> If we want to start documenting the binary format for all data types, it
> should go into the "Data Types" chapter. There, we already document the
> text format in great detail, so putting the binary format in there as
> well would make sense.
Agreed that the libpq manual is not the place for this, but I feel
like it will also be clutter in "Data Types". Perhaps we should
invent a new appendix or the like? Somewhere near the wire protocol
docs seems sensible.
regards, tom lane
On 01.11.22 09:15, Tom Lane wrote: > Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes: >> If we want to start documenting the binary format for all data types, it >> should go into the "Data Types" chapter. There, we already document the >> text format in great detail, so putting the binary format in there as >> well would make sense. > > Agreed that the libpq manual is not the place for this, but I feel > like it will also be clutter in "Data Types". Perhaps we should > invent a new appendix or the like? Somewhere near the wire protocol > docs seems sensible. Would that clutter the protocol docs? ;-) I suppose figuring out exactly where to put it and how to mark it up, etc., in a repeatable fashion is part of the job here.
Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
> On 01.11.22 09:15, Tom Lane wrote:
>> Agreed that the libpq manual is not the place for this, but I feel
>> like it will also be clutter in "Data Types". Perhaps we should
>> invent a new appendix or the like? Somewhere near the wire protocol
>> docs seems sensible.
> Would that clutter the protocol docs? ;-)
I said "near", not "in". At the time I was thinking "new appendix",
but I now recall that the wire protocol docs are not an appendix
but a chapter in the Internals division. So that doesn't seem like
quite the right place anyway.
Perhaps a new chapter under "IV. Client Interfaces" is the right
place?
If we wanted to get aggressive, we could move most of the nitpicky details
about datatype text formatting (e.g., the array quoting rules) there too.
I'm not set on that, but it'd make datatype.sgml smaller which could
hardly be a bad thing.
> I suppose figuring out exactly where to put it and how to mark it up,
> etc., in a repeatable fashion is part of the job here.
Yup.
regards, tom lane
On Thu, Nov 03, 2022 at 09:55:22AM -0400, Tom Lane wrote: > Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes: > > On 01.11.22 09:15, Tom Lane wrote: > >> Agreed that the libpq manual is not the place for this, but I feel > >> like it will also be clutter in "Data Types". Perhaps we should > >> invent a new appendix or the like? Somewhere near the wire protocol > >> docs seems sensible. > > > Would that clutter the protocol docs? ;-) > > I said "near", not "in". At the time I was thinking "new appendix", > but I now recall that the wire protocol docs are not an appendix > but a chapter in the Internals division. So that doesn't seem like > quite the right place anyway. > > Perhaps a new chapter under "IV. Client Interfaces" is the right > place? > > If we wanted to get aggressive, we could move most of the nitpicky details > about datatype text formatting (e.g., the array quoting rules) there too. > I'm not set on that, but it'd make datatype.sgml smaller which could > hardly be a bad thing. > > > I suppose figuring out exactly where to put it and how to mark it up, > > etc., in a repeatable fashion is part of the job here. > > Yup. I'll take a stab at adding a new chapter and share how that looks. Regards, Mark -- Mark Wong EDB https://enterprisedb.com
On Wed, Nov 09, 2022 at 12:47:23PM -0800, Mark Wong wrote: > I'll take a stab at adding a new chapter and share how that looks. This thread has stalled for three weeks, so I have marked it as returned with feedback. Once you have a new patch, please feel free to submit it. -- Michael
Attachment
On Thu, Nov 03, 2022 at 09:55:22AM -0400, Tom Lane wrote: > Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes: > > On 01.11.22 09:15, Tom Lane wrote: > >> Agreed that the libpq manual is not the place for this, but I feel > >> like it will also be clutter in "Data Types". Perhaps we should > >> invent a new appendix or the like? Somewhere near the wire protocol > >> docs seems sensible. > > > Would that clutter the protocol docs? ;-) > > I said "near", not "in". At the time I was thinking "new appendix", > but I now recall that the wire protocol docs are not an appendix > but a chapter in the Internals division. So that doesn't seem like > quite the right place anyway. > > Perhaps a new chapter under "IV. Client Interfaces" is the right > place? > > If we wanted to get aggressive, we could move most of the nitpicky details > about datatype text formatting (e.g., the array quoting rules) there too. > I'm not set on that, but it'd make datatype.sgml smaller which could > hardly be a bad thing. > > > I suppose figuring out exactly where to put it and how to mark it up, > > etc., in a repeatable fashion is part of the job here. > > Yup. How does this look? I've simply moved things around into a new "Binary Format" section with the few parts that I've started for some quick feedback about whether this is looking like the right landing place. Regards, Mark
Attachment
Mark Wong <markwkm@gmail.com> writes:
> On Thu, Nov 03, 2022 at 09:55:22AM -0400, Tom Lane wrote:
>> Perhaps a new chapter under "IV. Client Interfaces" is the right
>> place?
>>
>> If we wanted to get aggressive, we could move most of the nitpicky details
>> about datatype text formatting (e.g., the array quoting rules) there too.
>> I'm not set on that, but it'd make datatype.sgml smaller which could
>> hardly be a bad thing.
> I suppose figuring out exactly where to put it and how to mark it up,
> etc., in a repeatable fashion is part of the job here.
> How does this look?
> I've simply moved things around into a new "Binary Format" section with
> the few parts that I've started for some quick feedback about whether
> this is looking like the right landing place.
I took a quick look at this. The patch cannot be applied as given
because it seems to be editing binary-format.sgml from some prior
state, but that file doesn't exist at all yet. However, there's
enough visible in the patch to comment on.
Personally I'm not excited about supplying fragments of C code
like this. Those seem quite useless to non-C users, and I'm not
really sure that they'd be very helpful even for C users, because
you have to make a whole lot of assumptions about what the user
wants to do with the value. I think what *would* be helpful is
just plain prose documentation of the on-the-wire binary format.
I don't mind if you write something like
A float4 value is a 4-byte IEEE single-precision floating point
number. It is transmitted in network byte order, so you must
convert to local byte order. (C users can do this portably
using the standard ntohl() function.)
but I'm not sure an example is worth more than such a parenthetical
comment. Perhaps others disagree, though.
regards, tom lane
On Fri, Jan 20, 2023 at 12:58 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > I don't mind if you write something like > > A float4 value is a 4-byte IEEE single-precision floating point > number. It is transmitted in network byte order, so you must > convert to local byte order. (C users can do this portably > using the standard ntohl() function.) > > but I'm not sure an example is worth more than such a parenthetical > comment. Perhaps others disagree, though. I don't disagree with that. I do think that when you suggested documenting this rather than just adding some examples, you moved the goalposts a long way. If we're going to add this to the documentation, it probably ought to cover every data type we ship. Overall, I think that would be a better result than just adding a few examples for the most common data types to testlibpq*.c, but it's also substantially more work. I do agree with you that if we're going to document this, as opposed to provide examples, then a narrative style is more appropriate than a whole bunch of small sample programs; maintaining working code in the documentation seems like an annoying amount of maintenance and is probably not the most efficient way to communicate useful information. -- Robert Haas EDB: http://www.enterprisedb.com
On Fri, Jan 20, 2023 at 01:12:07PM -0500, Robert Haas wrote: > On Fri, Jan 20, 2023 at 12:58 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > > I don't mind if you write something like > > > > A float4 value is a 4-byte IEEE single-precision floating point > > number. It is transmitted in network byte order, so you must > > convert to local byte order. (C users can do this portably > > using the standard ntohl() function.) > > > > but I'm not sure an example is worth more than such a parenthetical > > comment. Perhaps others disagree, though. > > I don't disagree with that. > > I do think that when you suggested documenting this rather than just > adding some examples, you moved the goalposts a long way. If we're > going to add this to the documentation, it probably ought to cover > every data type we ship. Overall, I think that would be a better > result than just adding a few examples for the most common data types > to testlibpq*.c, but it's also substantially more work. I do agree > with you that if we're going to document this, as opposed to provide > examples, then a narrative style is more appropriate than a whole > bunch of small sample programs; maintaining working code in the > documentation seems like an annoying amount of maintenance and is > probably not the most efficient way to communicate useful information. Yeah, I will try to cover all the data types we ship. :) I'll keep at it and drop the code examples. Regards, Mark -- Mark Wong EDB: http://www.enterprisedb.com
On Mon, 23 Jan 2023 at 11:54, Mark Wong <markwkm@gmail.com> wrote: fficient way to communicate useful information. > > Yeah, I will try to cover all the data types we ship. :) I'll keep at > it and drop the code examples. I assume this isn't going to happen for this commitfest? If you think it is then shout otherwise I'll mark it Returned with Feedback and move it on to the next release. -- Gregory Stark As Commitfest Manager
On Mon, Mar 20, 2023 at 01:37:57PM -0400, Gregory Stark (as CFM) wrote: > On Mon, 23 Jan 2023 at 11:54, Mark Wong <markwkm@gmail.com> wrote: > fficient way to communicate useful information. > > > > Yeah, I will try to cover all the data types we ship. :) I'll keep at > > it and drop the code examples. > > I assume this isn't going to happen for this commitfest? If you think > it is then shout otherwise I'll mark it Returned with Feedback and > move it on to the next release. Sounds good. I actually thought I did that already, thanks for catching that. Regards, Mark -- Mark Wong EDB https://enterprisedb.com
> On 21 Mar 2023, at 14:44, Mark Wong <markwkm@gmail.com> wrote: > > On Mon, Mar 20, 2023 at 01:37:57PM -0400, Gregory Stark (as CFM) wrote: >> On Mon, 23 Jan 2023 at 11:54, Mark Wong <markwkm@gmail.com> wrote: >> fficient way to communicate useful information. >>> >>> Yeah, I will try to cover all the data types we ship. :) I'll keep at >>> it and drop the code examples. >> >> I assume this isn't going to happen for this commitfest? If you think >> it is then shout otherwise I'll mark it Returned with Feedback and >> move it on to the next release. > > Sounds good. I actually thought I did that already, thanks for catching > that. This has been waiting on author since January, unless there is a new patch ready I will close this as Returned with Feedback and you can resubmit for another CF when there is a new patch. -- Daniel Gustafsson