Re: group locking: incomplete patch, just for discussion - Mailing list pgsql-hackers

On Wed, Oct 29, 2014 at 10:21 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
> There is a real danger that your "ta-dah" moment sometime in the
> future contains flaws which need to be addressed, but we now have
> piles of questionable infrastructure lieing around. If you have
> similar doubts about anything I'm doing, please just ask.

I think the implication that the infrastructure created to date is of
questionable utility for parallelism is false and unjustifed.  Every
infrastructure patch thus far committed, and every patch proposed
other than this one, has been accompanied by code that demonstrates
what it does and how that's useful.  Most of them have had enough
interest by extension authors to generate bug reports from other
people - most notably Petr, who has obviously been playing with all of
the existing modules enough to notice bugs and come to conclusions
about the shortcomings of the existing technology similar to my own.
Interest in dynamic background workers seems to go considerably
further, to the point where it made 9.4's list of major features.

> If we could see the 6-12 patch descriptions and understand where you
> are going it would help.

OK, here's what I think we need (roughly):

1. The remaining pg_background patches.  The main things in there are
(a) the ability to propagate errors from a background worker back to
the user backend and (b) the ability to synchronize the background
worker's GUC state with the user backend.

2. Group locking.

3. Patches to synchronize the (a) snapshot, (b) combo CID hash, and
(c) transaction state from the user backend to a background worker.
Draft patches for (a) and (b) are already completed yet, but not
published because I didn't think anyone would be very excited about
them just yet.

4. A patch to add "parallel mode" in which a whole bunch of things are
prohibited in the user backend and even more things are prohibited in
the background worker - specifically, things that won't be safe in
those contexts.  Noah and Amit Kapila worked on this and we have draft
patches for this as well, also not published and for the same reasons.

5. A patch to add a "proisparallel" flag or similar so that we can
label parallel-safe functions.

6. A patch to add a way for a background worker that hits an error to
kick the user backend, probably via the ProcSignal mechanism, so that
the user doesn't have to poll the shm_mq structures it's using to
communicate with those backends for errors continuously (or
alternative sit idle while those background workers do all the work).
This might be something that can be skipped for early versions, but
eventually I think we'll need it.

7. Eventually, a dynamic shared memory allocator.  A lot of things can
be done without this, but some can't.

I think that's most of it.  No doubt we'll discover a few other needs
along the way, but if we had that infrastructure, then I think it
would be quite feasible for an experienced hacker to parallelize most
of what we want to parallelize without having to invent too much
that's really fundamental.

> If I did, I don't think it would be that useful to interfere.
> Cooperation seems better use of my time, if possible.

I am not asking anyone to interfere, in the sense of getting in the
way, but I certainly think it would be useful for people to contribute
code and/or reviews.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company