Re: Rewrite sinval messaging to reduce contention - Mailing list pgsql-patches

From Tom Lane
Subject Re: Rewrite sinval messaging to reduce contention
Date
Msg-id 12066.1213843039@sss.pgh.pa.us
Whole thread Raw
In response to Rewrite sinval messaging to reduce contention  (Tom Lane <tgl@sss.pgh.pa.us>)
Responses Re: Rewrite sinval messaging to reduce contention  (Tom Lane <tgl@sss.pgh.pa.us>)
List pgsql-patches
I wrote:
> In this thread back in January
> http://archives.postgresql.org/pgsql-performance/2008-01/msg00001.php
> we saw evidence that contention for SInvalLock was a problem,
> and we (or at least I) concluded that the sinval messaging system
> needed a nontrivial rewrite to get rid of its habit of sending everyone
> to read the queue at the same time.  Here is the proposed rewrite.

After further thought, I came up with two more improvements:

* It turns out not to be too hard to apply the idea of moving multiple
messages per LWLock acquisition on the writing side too, so I did that.

* There is not actually any good reason why writers have to block
readers or vice versa, except in the infrequent case that SICleanupQueue
is needed.  Writers don't look at the ProcStates that the readers are
changing.  The only point of overlap is that a writer will change
maxMsgNum which readers want to look at --- but we are already assuming
in several places that reading or writing an int is atomic, and if we
assume that here too, it seems to work fine.  To implement this, I split
SInvalLock into SInvalReadLock and SInvalWriteLock.

This seems to be reaching a point of diminishing returns for the
particular test case I'm using --- the TPS rate only went up from 700
to 730 or so.  However, enabling LWLOCK_STATS shows that the contention
rate on the sinval locks is now completely negligible --- one block
per thousand acquisitions on SInvalWriteLock, and less than one in
10000 on SInvalReadLock.  The vast majority of the LWLock contention
now comes from WALInsertLock and the LockMgr locks:

    Lock            # acquisitions    # times blocked

    SInvalReadLock        6469840        380
    SInvalWriteLock        240567        163
    WALInsertLock        2388805        89142
    LockMgr partition locks    8253142        177715

So I think this patch accomplishes the goal of making sinval not be a
cause of contention.

Patch version 2 attached.

            regards, tom lane

Attachment

pgsql-patches by date:

Previous
From: Simon Riggs
Date:
Subject: Re: [HACKERS] Hint Bits and Write I/O
Next
From: Tom Lane
Date:
Subject: Re: Rewrite sinval messaging to reduce contention