Re: CLOG contention, part 2 - Mailing list pgsql-hackers

<p><br /> On Feb 9, 2012 1:27 AM, "Robert Haas" <<a
href="mailto:robertmhaas@gmail.com">robertmhaas@gmail.com</a>><br /> > However, there is a potential fly in the
ointment:in other cases in<br /> > which we've reduced contention at the LWLock layer, we've ended up<br /> >
withvery nasty contention at the spinlock layer that can sometimes<br /> > eat more CPU time than the LWLock
contentiondid.   In that light, it<br /> > strikes me that it would be nice to be able to partition the<br /> >
contentionN ways rather than just 2 ways.  I think we could do that<br /> > as follows.  Instead of having one
controllock per SLRU, have N<br /> > locks, where N is probably a power of 2.  Divide the buffer pool for<br /> >
theSLRU N ways, and decree that each slice of the buffer pool is<br /> > controlled by one of the N locks.  Route
allrequests for a page P to<br /> > slice P mod N.  Unlike this approach, that wouldn't completely<br /> >
eliminatecontention at the LWLock level, but it would reduce it<br /> > proportional to the number of partitions,
andit would reduce spinlock<br /> > contention according to the number of partitions as well.  A down side<br />
>is that you'll need more buffers to get the same hit rate, but this<br /> > proposal has the same problem: it
doublesthe amount of memory<br /> > allocated for CLOG. <p>Splitting the SLRU into different parts is exactly the
sameapproach as associativity used in CPU caches. I found some numbers that analyze cache hit rate with different
associativities:<p><a
href="http://research.cs.wisc.edu/multifacet/misc/spec2000cache-data/">http://research.cs.wisc.edu/multifacet/misc/spec2000cache-data/</a><p>Now
obviouslyCPU cache access patterns are different from CLOG patterns, but I think that the numbers strongly suggest that
thereduction in hitrate might be less than what you fear. For example, the harmonic mean of data cache misses over all
benchmarkfor 16, 32 and 64 cache lines:<br /> | Size | Direct | 2-way LRU | 4-way LRU | 8-way LRU | Full LRU | <br />
|-------+-------------+-------------+-------------+-------------+-------------|<br /> | 1KB | 0.0863842-- | 0.0697167--
|0.0634309-- | 0.0563450-- | 0.0533706-- | <br /> | 2KB | 0.0571524-- | 0.0423833-- | 0.0360463-- | 0.0330364-- |
0.0305213--|<br /> | 4KB | 0.0370053-- | 0.0260286-- | 0.0222981-- | 0.0202763-- | 0.0190243-- |<p>As you can see, the
reductionin hit rate is rather small down to 4 way associative caches.<p>There may be a performance problem when
multipleCLOG pages that happen to sit in a single way become hot at the same time. The most likely case that I can come
upwith is multiple scans going over unhinted pages created at different time periods. If that is something to worry
about,then a tool that's used for CPUs is to employ a fully associative victim cache behind the main cache. If a CLOG
pageis evicted, it is transferred into the victim cache, evicting a page from there. When a page isn't found in the
maincache, the victim cache is first checked for a possible hit. The movement between the two caches doesn't need to
involveany memory copying - just swap pointers in metadata.<p>The victim cache will bring back concurrency issues when
thehit rate of the main cache is small - like the pgbench example you mentioned. In that case, a simple associative
cachewill allow multiple reads of clog pages simultaneously. On the other hand - in that case lock contention seems to
bethe symptom, rather than the disease. I think that those cases would be better handled by increasing the maximum CLOG
SLRUsize. The increase in memory usage should be a drop in the bucket for systems that have enough transaction
processingvelocity for that to be a problem. <p>--<br /> Ants Aasma