Re: Parallel WAL Archival Options - Mailing list pgsql-admin

From Ron
Subject Re: Parallel WAL Archival Options
Date
Msg-id a45c5157-e2be-8592-9447-c62d414fa3dc@gmail.com
Whole thread Raw
In response to Parallel WAL Archival Options  (Nikhil Shetty <nikhil.dba04@gmail.com>)
Responses Re: Parallel WAL Archival Options
List pgsql-admin
On 8/6/23 02:43, Nikhil Shetty wrote:
Hi Team,

I would like to know which backup&restore tools will be better for scenarios where the database is generating around 400 WALs per minute.

If my math is correct, 400x 16MB WAL files per minute is 400*(16*2^20)/60*8 / 10^6 = 895 MBits per second.  Plus overhead.

That's about 1Gbit/second.  Definitely nothing to sneeze at.

We are using wal-g but it is not able to keep pace with the wal generation. We increased the upload streams to 256 but no luck

Uploads to a remote server? 

Does wal-g compress files before sending them across the wire?  By how much?  Are you CPU or IO bound by having to compress that much data?

--
Born in Arizona, moved to Babylonia.

pgsql-admin by date:

Previous
From: Nikhil Shetty
Date:
Subject: Parallel WAL Archival Options
Next
From: "Michaeldba@sqlexec.com"
Date:
Subject: Re: Parallel WAL Archival Options