Re: Logical Replication Helpers WIP for discussion - Mailing list pgsql-hackers

On 15/12/14 19:42, Robert Haas wrote:
> On Mon, Dec 15, 2014 at 12:57 AM, Petr Jelinek <petr@2ndquadrant.com> wrote:
>> we've made few helper functions for making logical replication easier, I
>> bundled it into contrib module as this is mainly for discussion at this time
>> (I don't expect this to get committed any time soon, but it is good way to
>> iron out protocol, etc).
>>
>> I created sample logical decoding plugin that uses those functions and which
>> can be used for passing DML changes in platform/version independent
>> (hopefully) format.
>>
>> I will post sample apply BG worker also once I get some initial feedback
>> about this.
>>
>> It's hard to write tests for this as the binary changes contain transaction
>> ids and timestamps so the data changes constantly.
>>
>> This is of course based on the BDR work Andres, Craig and myself have been
>> doing.
>
> I can't understand, either from what you've written here or the rather
> sparse comments in the patch, what this might be good for.
>

What I tried to achieve here is to provide solution to many of the 
common problems faced by logical replication solutions. I believe the 
first step in designing the logical replication (now that we have the 
logical decoding) is making the output plugin and the efficient protocol 
so I started with that.

The code itself provides two main parts:
First is the lrep_utils common utility functions that solve things like 
transporting DML statements, and more importantly the changed data in 
efficient manner, trying to not do any conversion if not needed (when 
architecture/version matches) but falling back to binary/textual IO 
representation of individual types so that the cross platform/version 
replication works too. I think those should eventually end up in core 
(ie not in contrib) as they are helper functions likely to be shared by 
multiple extensions, but for now I keep them with the rest of the 
contrib module as I feel better experimenting inside that module.
There are also read functions that show how the other side could look 
like, but they are currently unused as the example apply worker is not 
part of the submission yet.

The second part is extensible output plugin which serves both as an 
example of the intended use of those common utility functions and also 
as actual working solution that can be used as base for several 
replication solutions.
It provides hooks for the replication solutions built on top of it that 
can be used for deciding if to replicate specific action on specific 
object and also injecting additional information to both BEGIN and 
COMMIT message - this can be useful for example when you are forwarding 
changes from another node and you wish to pass the information about the 
original node to the target one.

What I hope to get from this is agreement on the general approach and 
protocol so that we can have common base which will both make it easier 
to create external logical replication solutions and also eventually 
lead to full logical replication inside core PostgreSQL.

--  Petr Jelinek                  http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training &
Services