Re: bg worker: patch 1 of 6 - permanent process - Mailing list pgsql-hackers
From | Robert Haas |
---|---|
Subject | Re: bg worker: patch 1 of 6 - permanent process |
Date | |
Msg-id | AANLkTim3oYQW+O5J7ZWmJXiOy+VvQ=Baprxjcxu4W+Hf@mail.gmail.com Whole thread Raw |
In response to | Re: bg worker: patch 1 of 6 - permanent process (Markus Wanner <markus@bluegap.ch>) |
Responses |
Re: bg worker: patch 1 of 6 - permanent process
(Markus Wanner <markus@bluegap.ch>)
|
List | pgsql-hackers |
On Thu, Aug 26, 2010 at 6:07 AM, Markus Wanner <markus@bluegap.ch> wrote: > What'd be the benefits of having separate coordinator processes? They'd be > doing pretty much the same: coordinate background processes. (And yes, I > clearly consider autovacuum to be just one kind of background process). I dunno. It was just a thought. I haven't actually looked at the code to see how much synergy there is. (Sorry, been really busy...) >> I agree with this criticism, but the other thing that strikes me as a >> nonstarter is having the postmaster participate in the imessages >> framework. > > This is simply not the case (anymore). (And one of the reasons a separate > coordinator process is required, instead of letting the postmaster do this > kind of coordination). Oh, OK. I see now that I misinterpreted what you wrote. On the more general topic of imessages, I had one other thought that might be worth considering. Instead of using shared memory, what about using a file that is shared between the sender and receiver? So for example, perhaps each receiver will read messages from a file called pg_messages/%d, where %d is the backend ID. And writers will write into that file. Perhaps both readers and writers mmap() the file, or perhaps there's a way to make it work with just read() and write(). If you actually mmap() the file, you could probably manage it in a fashion pretty similar to what you had in mind for wamalloc, or some other setup that minimizes locking. In particular, ISTM that if we want this to be usable for parallel query, we'll want to be able to have one process streaming data in while another process streams data out, with minimal interference between these two activities. On the other hand, for processes that only send and receive messages occasionally, this might just be overkill (and overhead). You'd be just as well off wrapping the access to the file in an LWLock: the reader takes the lock, reads the data, marks it read, and releases the lock. The writer takes the lock, writes data, and releases the lock. It almost seems to me that there are two different kinds of messages here: control messages and data messages. Control messages are things like "vacuum this database!" or "flush your cache!" or "execute this query and send the results to backend %d!" or "cancel the currently executing query!". They are relatively small (in some cases, fixed-size), relatively low-volume, don't need complex locking, and can generally be processed serially but with high priority. Data messages are streams of tuples, either from a remote database from which we are replicating, or between backends that are executing a parallel query. These messages may be very large and extremely high-volume, are very sensitive to concurrency problems, but are not high-priority. We want to process them as quickly as possible, of course, but the work may get interrupted by control messages. Another point is that it's reasonable, at least in the case of parallel query, for the action of sending a data message to *block*. If one part of the query is too far ahead of the rest of the query, we don't want to queue up results forever, perhaps using CPU or I/O resources that some other backend needs to catch up, exhausting available disk space, etc.Instead, at some point, we just block and wait forthe queue to drain. I suppose there's no avoiding the possibility that sending a control message might also block, but certainly we wouldn't like a control message to block because the relevant queue is full of data messages. So I kind of wonder whether we ought to have two separate systems, one for data and one for control, with someone different characteristics. I notice that one of your bg worker patches is for OOO-messages. I apologize again for not having read through it, but how much does that resemble separating the control and data channels? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company
pgsql-hackers by date: