Re: Binary support for pgoutput plugin - Mailing list pgsql-hackers

From Dave Cramer
Subject Re: Binary support for pgoutput plugin
Date
Msg-id CADK3HH+tV-w3uk+suFVjX6wKUzUMGZzTAfd1LfLvrf1xVEhCkQ@mail.gmail.com
Whole thread Raw
In response to Re: Binary support for pgoutput plugin  (Petr Jelinek <petr.jelinek@2ndquadrant.com>)
Responses Re: Binary support for pgoutput plugin
List pgsql-hackers
Hi,


On Wed, 5 Jun 2019 at 07:18, Petr Jelinek <petr.jelinek@2ndquadrant.com> wrote:
Hi,

On 05/06/2019 00:08, Andres Freund wrote:
> Hi,
>
> On 2019-06-05 00:05:02 +0200, David Fetter wrote:
>> Would it make sense to work toward a binary format that's not
>> architecture-specific? I recall from COPY that our binary format is
>> not standardized across, for example, big- and little-endian machines.
>
> I think you recall wrongly. It's obviously possible that we have bugs
> around this, but output/input routines are supposed to handle a
> endianess independent format. That usually means that you have to do
> endianess conversions, but that doesn't make it non-standardized.
>

Yeah, there are really 3 formats of data we have, text protocol, binary
network protocol and internal on disk format. The internal on disk
format will not work across big/little-endian but network binary
protocol will.

FWIW I don't think the code for binary format was included in original
logical replication patch (I really tried to keep it as minimal as
possible), but the code and protocol is pretty much ready for adding that.
Yes, I looked through the public history and could not find it. Thanks for confirming. 

That said, pglogical has code which handles this (I guess Andres means
that by predecessor of pgoutput) so if you look for example at the
write_tuple/read_tuple/decide_datum_transfer in
https://github.com/2ndQuadrant/pglogical/blob/REL2_x_STABLE/pglogical_proto_native.c
that can help you give some ideas on how to approach this.

Thanks for the tip!


Dave Cramer


pgsql-hackers by date:

Previous
From: Petr Jelinek
Date:
Subject: Re: Binary support for pgoutput plugin
Next
From: Antonin Houska
Date:
Subject: Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3