Thread: Re: BUG #4053: libpq documentation should express clearly, that integers are passed in network octet order

Aleksej Saushev wrote:
> "Merlin Moncure" <mmoncure@gmail.com> writes:
>
> > Bruce Momjian wrote:
> >> This brings up a good question.  Exactly how do users know what format
> >> _binary_ is?  int4 is network byte order, but what about int8, float4,
> >> inet?
> >
> > This is exactly what libpqtypes solves.  Not only do we handle
> > formatting of binary formats, we provide a level of protection from
> > internal format changes for libpq users.  See the example here:
> > http://libpqtypes.esilo.com/.  So, documentation of binary formats
> > (including network byte ordering) are not required.
>
> No, it is still required. There's not a single reference to libpqtypes
> in Postgres documentation, and libpqtypes isn't part of the distribution,
> if I understand it right.

Agreed.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Aleksej Saushev wrote:
>> "Merlin Moncure" <mmoncure@gmail.com> writes:
>>
>>> Bruce Momjian wrote:
>>>> This brings up a good question.  Exactly how do users know what format
>>>> _binary_ is?  int4 is network byte order, but what about int8, float4,
>>>> inet?
>>> This is exactly what libpqtypes solves.  Not only do we handle
>>> formatting of binary formats, we provide a level of protection from
>>> internal format changes for libpq users.  See the example here:
>>> http://libpqtypes.esilo.com/.  So, documentation of binary formats
>>> (including network byte ordering) are not required.
>> No, it is still required. There's not a single reference to libpqtypes
>> in Postgres documentation, and libpqtypes isn't part of the distribution,
>> if I understand it right.
>
> Agreed.
>

Correct, libpqtypes is not part of the core.

What we are saying is, libpqtypes solves this problem.  I don't think
docs is the solution because that just supports the idea of libpq apps
directly handling binary formats; which makes changing these formats
MUCH harder in future releases.  Lots of libpq apps would be dependant
on these binary formats.

By all means, documenting this is probably a good idea.  I just don't
think it solves any of the interesting problems.

--
Andrew Chernow
eSilo, LLC
every bit counts
http://www.esilo.com/

Andrew Chernow <ac@esilo.com> writes:

> Bruce Momjian wrote:
>> Aleksej Saushev wrote:
>>> "Merlin Moncure" <mmoncure@gmail.com> writes:
>>>
>>>> Bruce Momjian wrote:
>>>>> This brings up a good question.  Exactly how do users know what format
>>>>> _binary_ is?  int4 is network byte order, but what about int8, float4,
>>>>> inet?
>>>> This is exactly what libpqtypes solves.  Not only do we handle
>>>> formatting of binary formats, we provide a level of protection from
>>>> internal format changes for libpq users.  See the example here:
>>>> http://libpqtypes.esilo.com/.  So, documentation of binary formats
>>>> (including network byte ordering) are not required.
>>> No, it is still required. There's not a single reference to libpqtypes
>>> in Postgres documentation, and libpqtypes isn't part of the distribution,
>>> if I understand it right.
>>
>> Agreed.
>
> Correct, libpqtypes is not part of the core.
>
> What we are saying is, libpqtypes solves this problem.  I don't
> think docs is the solution because that just supports the idea
> of libpq apps directly handling binary formats; which makes
> changing these formats MUCH harder in future releases.  Lots of
> libpq apps would be dependant on these binary formats.

Right, but that's upon developers to decide, if they're going to
support changing API. I'm pretty sure, I won't use it much anyway,
and internal documentation is good enough to fix the problem
in future.

> By all means, documenting this is probably a good idea.  I just
> don't think it solves any of the interesting problems.

The easiest way to address the lack of documentation is adding
reference to libpqtypes at appropriate place. Shipping libpqtypes
along is even better.

Andrew Chernow wrote:
> >>> This is exactly what libpqtypes solves.  Not only do we handle
> >>> formatting of binary formats, we provide a level of protection from
> >>> internal format changes for libpq users.  See the example here:
> >>> http://libpqtypes.esilo.com/.  So, documentation of binary formats
> >>> (including network byte ordering) are not required.
> >> No, it is still required. There's not a single reference to libpqtypes
> >> in Postgres documentation, and libpqtypes isn't part of the distribution,
> >> if I understand it right.
> >
> > Agreed.
> >
>
> Correct, libpqtypes is not part of the core.
>
> What we are saying is, libpqtypes solves this problem.  I don't think
> docs is the solution because that just supports the idea of libpq apps
> directly handling binary formats; which makes changing these formats
> MUCH harder in future releases.  Lots of libpq apps would be dependant
> on these binary formats.
>
> By all means, documenting this is probably a good idea.  I just don't
> think it solves any of the interesting problems.

Agreed that libpqtypes would solve the problem, but the odd thing to me
is that we have gotten very few requests for binary format information
since we added binary prepared parameters years ago, so is no one
actually using it?

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

On Thu, May 8, 2008 at 12:27 PM, Bruce Momjian <bruce@momjian.us> wrote:
> Andrew Chernow wrote:
>  > >>> This is exactly what libpqtypes solves.  Not only do we handle
>  > >>> formatting of binary formats, we provide a level of protection from
>  > >>> internal format changes for libpq users.  See the example here:
>  > >>> http://libpqtypes.esilo.com/.  So, documentation of binary formats
>  > >>> (including network byte ordering) are not required.
>  > >> No, it is still required. There's not a single reference to libpqtypes
>  > >> in Postgres documentation, and libpqtypes isn't part of the distribution,
>  > >> if I understand it right.
>  > >
>  > > Agreed.
>  > >
>  >
>  > Correct, libpqtypes is not part of the core.
>  >
>  > What we are saying is, libpqtypes solves this problem.  I don't think
>  > docs is the solution because that just supports the idea of libpq apps
>  > directly handling binary formats; which makes changing these formats
>  > MUCH harder in future releases.  Lots of libpq apps would be dependant
>  > on these binary formats.
>  >
>  > By all means, documenting this is probably a good idea.  I just don't
>  > think it solves any of the interesting problems.
>
>  Agreed that libpqtypes would solve the problem, but the odd thing to me
>  is that we have gotten very few requests for binary format information
>  since we added binary prepared parameters years ago, so is no one
>  actually using it?

libpqtypes is safe to use right now if you are willing to patch libpq
yourself back to at least 8.2.  back to 7.4 is possible but some
internal formats may have changes (we didn't check back that far) so
you'd have to be a little careful.

If I searched the archives, I could probably come up with several
requests for documentation/explanations on how to use the binary
format, with at least one or two other patches to do byteswapping.
Since libpqtypes automatically parameterizes your query and handles
all the details of byteswapping, why would you ever want to do it
another way?  What possible advantage would there be to ever invoking
PQexecParams manually? (PQexec has a small speed advantage for simple
queries, so that at least still has a place).

Add this to the fact that you now have a clean way to handle things
like arrays and composites without all the tedious and error prone
text parsing.  I understand arguments related to code maintenance and
the like, but for libpq developers, there would never be a reason
(outside of occasional PQexec), to run a non 'params' based query
because it's faster, easier, and safer.

It's a catch-22 here...a lot of people are simply unaware of what the
postgresql binary flag is even supposed to do, since it's so poorly
documented.  By the way, even with documentation, any app relying on
binary formats without the protection of libpqtypes could suddenly and
spectacularly break since there is zero protection from binary format
changes.

merlin

Bruce Momjian wrote:
>
> Agreed that libpqtypes would solve the problem, but the odd thing to me
> is that we have gotten very few requests for binary format information
> since we added binary prepared parameters years ago, so is no one
> actually using it?
>

I think part of the answer is lack of documentation and possibly how
tedious of an API call PQexecParams can be.  If anything, using the
parameterized method promotes much better "safer" coding practices, but
currently its at quite a programming effort cost.

--
Andrew Chernow
eSilo, LLC
every bit counts
http://www.esilo.com/