If speed create problem than make 3 easy script. 1. compress wal_archive. 2. ship wal archive over network and than 3. in the secondary server just decompress it.
Appreciate the responses, the network is good.. around 350 MB/s between prinary and secondary, both primary and secondary in the same DC. And it’s ASYC replication..
I see the issue when the writes are super heavy ,
I am only thinking if we can add more processes that sends / receives the WALs , we can speed up the transfer rate of WALs to the secondary..
> On Sep 4, 2024, at 9:03 PM, Karthik Yellapragada <karthik.yellapragada@gmail.com> wrote: > > Hello all, > > I frequently face the problem of wals generated faster than the wals transferred and applied. > > Is there a way to speed up the process? So I don’t accumulate a lot of WALs at the primary? > > Regards, > Karthik.
What do you mean by parallelized WAL sender? Do you have many replicas connecting to the primary? Have you tried cascading replication?
What’s the network between the systems? Is a WAN with high latency? Can the replica pull from the WAL file from an archive? For a high latency networks like New York to London; I would terminate the WAL receiver via a script if replication exceeded an acceptable delay to ensure SLA where met. It is faster to pull the WAL files from the local archive which where already replicated to London than streaming replication over a high latency WAN. After It catches up, it will reconnect to streaming replication.
