Thread: [HACKERS] HACKERS[PROPOSAL] split ProcArrayLock into multiple parts

On Tue, Jun 6, 2017 at 1:00 AM, Jim Van Fleet <vanfleet@us.ibm.com> wrote:
> Hi,
>
> I have been experimenting with splitting  the ProcArrayLock into parts.
> That is, to Acquire the ProcArrayLock in shared mode, it is only necessary
> to acquire one of the parts in shared mode; to acquire the lock in exclusive
> mode, all of the parts must be acquired in exclusive mode. For those
> interested, I have attached a design description of the change.
>
> This approach has been quite successful on large systems with the hammerdb
> benchmark.With a prototype based on 10 master source and running on power8
> (model 8335-GCA with 2sockets, 20 core)
>  hammerdb  improved by 16%; On intel (Intel(R) Xeon(R) CPU E5-2699 v4 @
> 2.20GHz, 2 socket, 44 core) with 9.6 base and prototype hammerdb improved by
> 4%. (attached is a set of spreadsheets for power8.
>
> The down side is that on smaller configurations (single socket) where there
> is less "lock thrashing" in the storage subsystem and there are multiple
> Lwlocks to take for an exclusive acquire, there is a decided downturn in
> performance. On  hammerdb, the prototype was 6% worse than the base on a
> single socket power configuration.
>

I think any patch having 6% regression on one machine configuration
and 16% improvement on another machine configuration is not a net win.
However, if there is a way to address the regression, then it will
look much attractive.

> If there is interest in this approach, I will submit a patch.
>

The basic idea is clear from your description, but it will be better
if you share the patch as well.  It will not only help people to
review and provide you feedback but also allow them to test and see if
they can reproduce the numbers you have mentioned in the mail.

There is some related work which was previously proposed in this area
("Cache the snapshot") [1] and it claims to reduce contention around
ProcArrayLock.  I am not sure if that patch still applies, however, if
you find it relevant and you are interested in evaluating the same,
then we can request the author to post a rebased version if it doesn't
apply.

[1] - https://www.postgresql.org/message-id/CAD__OuiwEi5sHe2wwQCK36Ac9QMhvJuqG3CfPN%2BOFCMb7rdruQ%40mail.gmail.com

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

On Wed, Jun 7, 2017 at 12:29 PM, Jim Van Fleet <vanfleet@us.ibm.com> wrote:
>> The basic idea is clear from your description, but it will be better
>> if you share the patch as well.  It will not only help people to
>> review and provide you feedback but also allow them to test and see if
>> they can reproduce the numbers you have mentioned in the mail.
>
> OK -- would love the feedback and any suggestions on how to mitigate the low
> end problems.

Did you intend to attach a patch?

> Sokolov Yura has a patch which, to me, looks good for pgbench rw
> performance.  Does not do so well with hammerdb (about the same as base) on
> single socket and two socket.

Any idea why?  I think we will have to understand *why* certain things
help in some situations and not others, not just *that* they do, in
order to come up with a good solution to this problem.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Good day Robert, Jim, and everyone.

On 2017-06-08 00:06, Jim Van Fleet wrote:
> Robert Haas <robertmhaas@gmail.com> wrote on 06/07/2017 12:12:02 PM:
> 
>> > OK -- would love the feedback and any suggestions on how to
> mitigate the low
>> > end problems.
>> 
>> Did you intend to attach a patch?
> Yes I do -- tomorrow or Thursday -- needs a little cleaning up ...
> 
>> > Sokolov Yura has a patch which, to me, looks good for pgbench rw
>> > performance.  Does not do so well with hammerdb (about the same as
> base) on
>> > single socket and two socket.
>> 
>> Any idea why?  I think we will have to understand *why* certain
> things
>> help in some situations and not others, not just *that* they do, in
>> order to come up with a good solution to this problem.

My patch improves acquiring contended/blocking LWLock on NUMA cause:
a. patched procedure generates much lesser writes, especially because  taking WaitListLock is unified with acquiring
thelock itself.  Access to modified memory is very expensive on NUMA, so less writes  leads to less wasted time.
 
b. it spins several time on lock->state in attempts to acquire lock  before starting attempts to queue self to wait
list.It is really the  cause of some speedup. Without spinning patch just removes  degradation on contention.  I don't
knowwhy spinning doesn't improves single socket performance  though :-) Probably still because all algorithmic overhead
(system calls, sleeping and awakening process) is not too expensive until  NUMA is involved.
 

> Looking at the data now -- LWLockAquire philosophy is different -- at
> first glance I would have guessed "about the same" as the base, but I
> can not yet explain why we have super pgbench rw performance and "the
> same" hammerdb performance

My patch improves only blocking contention, ie when a lot of EXCLUSIVE
locks are involved. pgbench rw generates a lot of write traffic, so
there is a lot of contention and waiting on WALInsertLocks (in
XLogInsertRecord, and waiting in XLogFlush), WalWriteLock (in
XLogFlush), CLogControlLock (in TransactionIdSetTreeStatus).

The case when SHARED lock is much more common than EXCLUSIVE is not
affected by patch, because SHARED is acquired then on the fast path
in both original and patched version.

So, looks like hammerdb doesn't produce much EXCLUSIVE contention on
LWLocks, so it is not improved with the patch.

Splitting ProcArrayLock helps with acquiring SHARED lock on NUMA in
absence of EXCLUSIVE lock because of the same reason why my patch
improves acquiring of blocking lock: less writes to same memory.
Since every process writes to some one part of ProcArrayLock, there
is a lot less writes to each part of ProcArrayLock, so acquiring
SHARED lock pays lesser for accessing to modified memory on NUMA.

Probably I'm mistaken somewhere.

> 
>> 
>> --
>> Robert Haas
>> EnterpriseDB: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>> 

-- 
Sokolov Yura aka funny_falcon
Postgres Professional: https://postgrespro.ru
The Russian Postgres Company