Thread: Parallelize WAL Sender and WAL receiver

> On Sep 4, 2024, at 9:03 PM, Karthik Yellapragada <karthik.yellapragada@gmail.com> wrote:
>
> Hello all,
>
> I frequently face the problem of wals generated faster than the wals transferred and applied.
>
> Is there a way to speed up the process? So I don’t accumulate a lot of WALs at the primary?
>
> Regards,
> Karthik.

What do you mean by parallelized WAL sender? Do you have many replicas connecting to the primary? Have you tried
cascadingreplication? 

What’s the network between the systems? Is a WAN with high latency? Can the replica pull from the WAL file from an
archive? For a high latency networks like New York to London; I would terminate the WAL receiver via a script if
replicationexceeded an acceptable delay to ensure SLA  where met.  It is faster to pull the WAL files from the local
archivewhich where already replicated to London than streaming replication over a high latency WAN.  After It catches
up,it will reconnect to streaming replication.   

> On Sep 4, 2024, at 11:47 PM, Karthik Yellapragada <karthik.yellapragada@gmail.com> wrote:
>
> Appreciate the responses, the network is good.. around 350 MB/s between prinary and secondary, both primary and
secondaryin the same DC.  And it’s ASYC replication..  
>
> I see the issue when the writes are super heavy ,
> I am only thinking if we can add more processes that sends / receives the WALs , we can speed up the transfer rate of
WALsto the secondary.. 
>
> I understand the Apply can not be parallelized..

Sounds like you have 10GbE between the nodes? Are you sure it’s send rate? If so, I would look at the disk subsystem.

> On Sep 5, 2024, at 1:54 AM, ajit wangkhem <ajit.wangkhem@gmail.com> wrote:
>
> If speed create problem than make 3 easy script. 1. compress wal_archive. 2. ship wal archive over network and than
3.in the secondary server just decompress it.  

I'm guessing you've never benchmarked your compression? With 10GbE, the CPU overhead of compression will just slow it
down.

On Fri, 2024-09-06 at 08:33 +0500, Muhammad Usman Khan wrote:
> On Thu, 5 Sept 2024 at 06:04, Karthik Yellapragada <karthik.yellapragada@gmail.com> wrote:
> > I frequently face the problem of wals generated faster than the wals transferred and applied.
> >
> > Is there a way to speed up the process? So I don’t accumulate a lot of WALs at the primary?
> >
>
> You can try to set the following parameters in the conf file which may help to overcome this problem.
>
> wal_sender_timeout = 60s
> wal_receiver_status_interval = 1s
> Ensure you have sufficient WAL senders  like max_wal_senders = 10
> max_wal_size = 2GB
> min_wal_size = 80MB
> wal_buffers = 16MB
> wal_level = replica

These settings won't do anything to mitigate the problem.

All you can do is get more network bandwidth.

Yours,
Laurenz Albe