Thread: CDC feature request

CDC feature request

From
Stepan Yankevych
Date:

Hi All!

 

Hopefully I’m using correct mail list  

If not please show me right direction 😊

 

I’m quite struggling without native Change Data Capture feature in PostgreSQL.

 

That would be great to implement it, possibly in not so complicated way.

 

Can Logical replication be a little be modified or reused to do not replicate data into destination table as is but to insert each change into “change table” (like in oracle 11 CDC)?

That change table should have at lease few additional columns

  • Operation (I/D/U)
  • txid
  • Commit_time_stamp

 

Thanks!

 

Stepan Yankevych

Re: CDC feature request

From
Magnus Hagander
Date:


On Thu, Mar 18, 2021 at 2:03 PM Stepan Yankevych <Stepan_Yankevych@epam.com> wrote:

Hi All!

 

Hopefully I’m using correct mail list  

If not please show me right direction 😊

 

I’m quite struggling without native Change Data Capture feature in PostgreSQL.

 

That would be great to implement it, possibly in not so complicated way.

 

Can Logical replication be a little be modified or reused to do not replicate data into destination table as is but to insert each change into “change table” (like in oracle 11 CDC)?

That change table should have at lease few additional columns

  • Operation (I/D/U)
  • txid
  • Commit_time_stamp

If you look at logical decoding, that's basically what you have, isn't it? It won't go into a table, but you can consume it into one if you want. Look at for example wal2json for examples of how to consume it -- but the system is pluggable so you can build your own or use one of the others available plugins.
 
--

Re: CDC feature request

From
Ronan Dunklau
Date:
>> 
>> That change table should have at lease few additional columns
>> 
>> * Operation (I/D/U)
>> * txid
>> * Commit_time_stamp
> 
> If you look at logical decoding, that's basically what you have, isn't
> it? It won't go into a table, but you can consume it into one if you
> want. Look at for example wal2json for examples of how to consume it
> -- but the system is pluggable so you can build your own or use one of
> the others available plugins.

Hello,

At my work, we basically did that, using wal2json, here: 
https://github.com/peopledoc/connemara/blob/master/connemara_replication/src/connemara_replication.c

The code is quite simple, you could probably inspire yourself from that, 
or even use it directly if your needs are basic and match was is 
offered. The replication code was written to be as fast and simple as 
possible.



Re: CDC feature request

From
Álvaro Hernández
Date:


On 18/3/21 14:03, Stepan Yankevych wrote:

Hi All!

 

Hopefully I’m using correct mail list  

If not please show me right direction 😊

 

I’m quite struggling without native Change Data Capture feature in PostgreSQL.

 

That would be great to implement it, possibly in not so complicated way.

 

Can Logical replication be a little be modified or reused to do not replicate data into destination table as is but to insert each change into “change table” (like in oracle 11 CDC)?

That change table should have at lease few additional columns

  • Operation (I/D/U)
  • txid
  • Commit_time_stamp

 

Thanks!

 

Stepan Yankevych


    Hi Stepan.

    I would recommend you to check https://debezium.io/, it stores every change in Kafka with detailed metadata, and you can later transform and/or inject it into any destination, with great level of flexibility, using any of the database connectors available.

   
    Álvaro

-- 

Alvaro Hernandez


-----------
OnGres