Re: FlexLocks - Mailing list pgsql-hackers
From | Pavan Deolasee |
---|---|
Subject | Re: FlexLocks |
Date | |
Msg-id | CABOikdMqH6hePzvAGtWf37TWO-OpmaeX7zQhHYYbV7N4e_1zSQ@mail.gmail.com Whole thread Raw |
In response to | Re: FlexLocks (Pavan Deolasee <pavan.deolasee@gmail.com>) |
Responses |
Re: FlexLocks
|
List | pgsql-hackers |
On Thu, Nov 17, 2011 at 10:19 AM, Pavan Deolasee <pavan.deolasee@gmail.com> wrote: > On Thu, Nov 17, 2011 at 10:01 AM, Robert Haas <robertmhaas@gmail.com> wrote: > >> >> I am not convinced that that's a better API. I mean, consider >> something like this: >> >> /* >> * OK, let's do it. First let other backends know I'm in ANALYZE. >> */ >> LWLockAcquire(ProcArrayLock, LW_EXCLUSIVE); >> MyProc->vacuumFlags |= PROC_IN_ANALYZE; >> LWLockRelease(ProcArrayLock); > >> I'm not sure exactly how you'd proposed to rewrite that, but I think >> it's almost guaranteed to be more than three lines of code. > > I would guess the ReqRes will look something like this where > ReqResRequest/Response would probably be union of all various requests > and responses, one for each type of request: > > struct ReqRes { > ReqResRequestType reqtype; > ReqResRequest req; > ReqResResponse res; > } > > The code above can be rewritten as: > > reqRes.reqtype = RR_PROC_SET_VACUUMFLAGS; > reqRes.req.set_vacuumflags.flags = PROC_IN_ANALYZE; > LWLockExecute(ProcArrayLock, LW_EXCLUSIVE, &reqRes); > My apologies for hijacking the thread, but the work seems quite related, so I thought I should post here instead of starting a new thread. Here is a WIP patch based on the idea of having a shared Q. A process trying to access the shared memory protected by a LWLock, sets up the task in its PGPROC and calls a new API LWLockExecute(). If the LWLock is available, the task is performed immediately and the function returns. Otherwise, the process queues up itself on the lock. When the last shared lock holder or the exclusive lock holder call LWLockRelease(), it scans through such pending tasks, executes them via a callback mechanism and wakes all those processes along with any other normal waiter(s) waiting on LWLockAcquire(). I have only coded for ProcArrayEndTransaction, but it should fairly easy to extend the usage at some more places, especially those which does some simple modifications to the protected area. I don't propose to use the technique for every user of LWLock, but there can be some obvious candidates, including this one that Robert found out. I see 35-40% improvement for 32-80 clients on a 5 minutes pgbench -N run with scale factor of 100 and permanent tables. This is on a 32-core HP IA box. There are few things that need some deliberations. The pending tasks are right now executed while holding the mutex (spinlock). This is good and bad for obvious reasons. We can possibly change that so that the work is done without holding the spinlock or leave to the caller to choose the behavior. Doing it without holding the spinlock will make the technique interesting for many more callers. We can also rework the task execution so that pending similar requests from multiple callers can be combined and executed with a single callback, if the caller knows its safe to do so. I haven't thought through the API/callback changes to support that, but its definitely possible and could be quite useful in many cases. For example, status of many transactions can be checked with a single lookup of the ProcArray. Or WAL inserts from multiple processes can be combined and written at once. Thanks, Pavan -- Pavan Deolasee EnterpriseDB http://www.enterprisedb.com
Attachment
pgsql-hackers by date: