RE: Perform streaming logical transactions by background workers and parallel apply - Mailing list pgsql-hackers

From Hayato Kuroda (Fujitsu)
Subject RE: Perform streaming logical transactions by background workers and parallel apply
Date
Msg-id TYAPR01MB5866160DE81FA2D88B8F22DEF5159@TYAPR01MB5866.jpnprd01.prod.outlook.com
Whole thread Raw
In response to Re: Perform streaming logical transactions by background workers and parallel apply  (Amit Kapila <amit.kapila16@gmail.com>)
List pgsql-hackers
Dear hackers,

> 1. I think testing the scenario where the shm_mq buffer is full
> between the leader and parallel apply worker would require a large
> amount of data and then also there is no guarantee. How about having a
> developer GUC [1] force_apply_serialize which allows us to serialize
> the changes and only after commit the parallel apply worker would be
> allowed to apply it?
> 
> I am not sure if we can reliably test the serialization of partial
> changes (like some changes have been already sent to parallel apply
> worker and then serialization happens) but at least we can test the
> serialization of complete xacts and their execution via parallel apply
> worker.

I agreed for adding the developer options, because the part that LA serialize
changes and PAs read and apply them might be complex. I have reported some
bugs around here.

One idea: A threshold(integer) can be introduced as the developer GUC.
LA skips to send data or jumps to serialization part to PA via shm_mq_send() when
it has sent more than (threshold) times. This may be able to test the partial-serialization case.
Default(-1) means no-op, and 0 means all changes must be serialized.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED


pgsql-hackers by date:

Previous
From: Dean Rasheed
Date:
Subject: Re: Non-decimal integer literals
Next
From: Bharath Rupireddy
Date:
Subject: Re: Introduce a new view for checkpointer related stats