Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes - Mailing list pgsql-hackers

From SATYANARAYANA NARLAPURAM
Subject Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes
Date
Msg-id CAHg+QDfVt90zLuk=TtbfoMA=xNd5Rr5P++r9v-B--ZyXqbEDtw@mail.gmail.com
Whole thread Raw
In response to Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes  (Dilip Kumar <dilipbalaut@gmail.com>)
List pgsql-hackers


On Thu, Dec 30, 2021 at 12:20 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Thu, Dec 30, 2021 at 1:41 PM Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote:

>
> Yeah, that's true, but even if we are blocking the transactions from committing then also it is possible that a new connection can come and generate more WAL,  yeah but I agree with the other part that if you throttle after committing then the user can cancel the queries and generate more WAL from those sessions as well.  But that is an extreme case where application developers want to bypass the throttling and want to generate more WALs.

How about having the new hook at the start of the new txn?  If we do
this, when the limit for the throttling is exceeded, the current txn
(even if it is a long running one) continues to do the WAL insertions,
the next txns would get blocked. Thoughts?

Do you mean while StartTransactionCommand or while assigning a new transaction id? If it is at StartTransactionCommand then we would be blocking the sessions which are only performing read queries right? 

Definitely not at StartTransactionCommand but possibly while assigning transaction Id inAssignTransactionId. Blocking readers is never the intent.
 
If we are doing at the transaction assignment level then we might be holding some of the locks so this might not be any better than throttling inside the commit.

If we define RPO as no transaction can commit when the wal_distance is more than configured MB, we had to throttle the writes before committing the transaction and new WAL generation by new connections or active doesn't matter as the transactions can't be committed and visible to the user. If the RPO is defined as no new write transactions allowed when wal_distance > configured MB, then we can block assigning the new transaction IDs until the RPO policy is met. IMHO, following the sync replication semantics is easier and more explainable as it is already familiar to the customers.
 

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

pgsql-hackers by date:

Previous
From: Bharath Rupireddy
Date:
Subject: Re: warn if GUC set to an invalid shared library
Next
From: "wangw.fnst@fujitsu.com"
Date:
Subject: RE: row filtering for logical replication