Re: Challenges with huge postgresql deployments in 300+ TBs - Mailing list pgsql-general

From Manu
Subject Re: Challenges with huge postgresql deployments in 300+ TBs
Date
Msg-id 179156909812.373782.3765533734963931820@gmail.com
Whole thread
In response to Re: Challenges with huge postgresql deployments in 300+ TBs  (Adrian Klaver <adrian.klaver@aklaver.com>)
List pgsql-general
Hi Kavya,

> 3. Fixing a standby that lags a lot

Of the operational walls, this is the one I would size before
committing rather than after, so I put together a small harness that
turns it into a number: it loads indexed data, pauses replay to
separate WAL generation from replay, then times pure replay from local
WAL and reports MB/s with a 95% CI and a lag curve.  About 15 minutes
on a couple hundred MB, and it refuses assert-enabled builds so the
timing is real:

  https://github.com/Manuelreyesbravo/standby-replay-ceiling

The numbers are hardware-specific -- one laptop core replays around
190 MB/s here, CPU-bound.  The point is to run it on a subset of your
intended storage and CPU and get your own ceiling before building the
full system, which matches Adrian's start-from-a-test-setup
suggestion; whether one replay core keeps up then depends on how much
WAL your write workload generates.

It measures that wall, it does not move it.  Happy to adjust it if your
setup would test differently.

Regards,
Manu



pgsql-general by date:

Previous
From: Adrian Klaver
Date:
Subject: Re: Challenges with huge postgresql deployments in 300+ TBs