Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes - Mailing list pgsql-hackers

From SATYANARAYANA NARLAPURAM
Subject Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes
Date
Msg-id CAHg+QDf8RPOiAb2Qs0f1y6DCG3xchu39bSg6GwXUUbC12+4aMQ@mail.gmail.com
Whole thread Raw
In response to Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes  (Dilip Kumar <dilipbalaut@gmail.com>)
Responses Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes
Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes
List pgsql-hackers


On Sat, Dec 25, 2021 at 9:25 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Sun, Dec 26, 2021 at 10:36 AM SATYANARAYANA NARLAPURAM <satyanarlapuram@gmail.com> wrote:

Actually all the WAL insertions are done under a critical section (except few exceptions), that means if you see all the references of XLogInsert(), it is always called under the critical section and that is my main worry about hooking at XLogInsert level.

Got it, understood the concern. But can we document the limitations of the hook and let the hook take care of it? I don't expect an error to be thrown here since we are not planning to allocate memory or make file system calls but instead look at the shared memory state and add delays when required.


Yet another problem is that if we are in XlogInsert() that means we are holding the buffer locks on all the pages we have modified, so if we add a hook at that level which can make it wait then we would also block any of the read operations needed to read from those buffers.  I haven't thought what could be better way to do this but this is certainly not good.

Yes, this is a problem. The other approach is adding a hook at XLogWrite/XLogFlush? All the other backends will be waiting behind the WALWriteLock. The process that is performing the write enters into a busy loop with small delays until the criteria are met. Inability to process the interrupts inside the critical section is a challenge in both approaches. Any other thoughts?
 

 
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

pgsql-hackers by date:

Previous
From: Masahiko Sawada
Date:
Subject: Re: Allow escape in application_name
Next
From: Tomas Vondra
Date:
Subject: Re: sequences vs. synchronous replication