Re: Speed up Clog Access by increasing CLOG buffers - Mailing list pgsql-hackers

From Simon Riggs
Subject Re: Speed up Clog Access by increasing CLOG buffers
Date
Msg-id CANP8+j+q4+ZP0JExgvDRPBpW6cDjb15nvjBR3iWmyoHH--3dCg@mail.gmail.com
Whole thread Raw
In response to Re: Speed up Clog Access by increasing CLOG buffers  (Amit Kapila <amit.kapila16@gmail.com>)
Responses Re: Speed up Clog Access by increasing CLOG buffers  (Amit Kapila <amit.kapila16@gmail.com>)
List pgsql-hackers
On 17 November 2015 at 11:27, Amit Kapila <amit.kapila16@gmail.com> wrote:
 
Attached patch group_update_clog_v1.patch
implements this idea.

I don't think we should be doing this only for transactions that don't have subtransactions.

The reason for not doing this optimization for subtransactions is that we
need to advertise the information that Group leader needs for updating
the transaction status and if we want to do it for sub transactions, then
all the subtransaction id's needs to be advertised.  Now here the tricky
part is that number of subtransactions for which the status needs to
be updated is dynamic, so reserving memory for it would be difficult.
However, we can reserve some space in Proc like we do for XidCache
(cache of sub transaction ids) and then use that to advertise that many
Xid's at-a-time or just allow this optimization if number of subtransactions
is lesser than or equal to the size of this new XidCache.  I am not sure
if it is good idea to use the existing XidCache for this purpose in which
case we need to have a separate space in PGProc for this purpose.  I
don't see allocating space for 64 or so subxid's as a problem, however
doing it for bigger number could be cause of concern.  
 
We are trying to speed up real cases, not just benchmarks.

So +1 for the concept, patch is going in right direction though lets do the full press-up.


I have mentioned above the reason for not doing it for sub transactions, if
you think it is viable to reserve space in shared memory for this purpose, then
I can include the optimization for subtransactions as well.

The number of subxids is unbounded, so as you say, reserving shmem isn't viable.

I'm interested in real world cases, so allocating 65 xids per process isn't needed, but we can say is that the optimization shouldn't break down abruptly in the presence of a small/reasonable number of subtransactions.

--
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

pgsql-hackers by date:

Previous
From: Amit Kapila
Date:
Subject: Re: Speed up Clog Access by increasing CLOG buffers
Next
From: Vladimir Borodin
Date:
Subject: Re: RFC: replace pg_stat_activity.waiting with something more descriptive