Thread: how to pass data (tuples) to worker processes?
Hi, I'm learning how to use the "background worker processes" commited in 9.3. The usage basics are quite nicely illustrated in the worker_spi extension (kudos to those who designed the feature / extension). I'm not quite sure how to pass data between the regular backend and a worker. Implementing the channel (socket/pipe/...) itself is not a big deal, that's IPC 101, but deciding which data to copy (and how) is. Say I need to forward a tuple to the worker process - e.g. from a nodeAgg node, so that the worker can build the hash table. Is there something (a rule of a thumb, method, ...) that would help me to identify the pieces of data that need to be copied? Or do I need to do the go through the objects and decide what to copy and how on my own? regards Tomas
Tomas Vondra wrote: > I'm learning how to use the "background worker processes" commited in > 9.3. The usage basics are quite nicely illustrated in the worker_spi > extension (kudos to those who designed the feature / extension). Thanks! > I'm not quite sure how to pass data between the regular backend and a > worker. Implementing the channel (socket/pipe/...) itself is not a big > deal, that's IPC 101, but deciding which data to copy (and how) is. > > Say I need to forward a tuple to the worker process - e.g. from a > nodeAgg node, so that the worker can build the hash table. Is there > something (a rule of a thumb, method, ...) that would help me to > identify the pieces of data that need to be copied? > > Or do I need to do the go through the objects and decide what to copy > and how on my own? Were you able to figure it out? If so, would you share? -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Sat, Aug 3, 2013 at 5:43 AM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
Tomas Vondra wrote:
> I'm learning how to use the "background worker processes" commited in
> 9.3. The usage basics are quite nicely illustrated in the worker_spi
> extension (kudos to those who designed the feature / extension).
Thanks!
> I'm not quite sure how to pass data between the regular backend and a
> worker. Implementing the channel (socket/pipe/...) itself is not a big
> deal, that's IPC 101, but deciding which data to copy (and how) is.
>
> [...]
Were you able to figure it out? If so, would you share?
I'm also in the middle of doing some experiments with bgworkers, and for me it's the IPC part that's proving tricky. I'd love to have a simple socket that can be used to communicate with the bgworker. But because the bgworker is launched by the postmaster -- and not the backend which registers it -- there's no chance for the bgworker to inherit one end of the socketpair().
Tomas: in the end, what approach did you use for IPC?
Robert: any chance you could share a few more details on the enhancements you're planning for bgworkers? I seem to recall reading that communicating with the dynamic bgworkers after they had been launched was next on your agenda...
Regards,
Andrew Tipton
On Sat, Aug 3, 2013 at 6:31 AM, Andrew Tipton <andrew@kiwidrew.com> wrote: > Robert: any chance you could share a few more details on the enhancements > you're planning for bgworkers? I seem to recall reading that communicating > with the dynamic bgworkers after they had been launched was next on your > agenda... Yeah, it is. I'm working on a patch to allow additional shared memory segments to be created on the fly. The idea I'm working with is that a backend that plans to launch a worker will first create a dynamic shared memory segment, then pass the ID of that segment to the worker via bgw_main_arg. The worker will map the segment, and then the two processes can use that to communicate. My thought is to create a queue abstraction that sits on top of the dynamic shared memory infrastructure, so that you can set aside a portion of your dynamic shared memory segment to use as a ring buffer and send messages back and forth with using some kind of API along these lines: extern void dsm_queue_send(dsm_queue *, char *data, uint64 len); extern uint64 dsm_queue_receive(dsm_queue *, char **dataptr); It would also be possible to implement message sending and receiving using pipes, but I'm leaning away from that because it would require even more OS-dependent code than I'm already having to write, and writing OS-dependent shim layers is one of the world's less-rewarding coding tasks; and also because I think it will be easier to achieve zero-copy semantics using shared memory. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Tuesday, August 06, 2013 6:29 PM Robert Haas wrote: > On Sat, Aug 3, 2013 at 6:31 AM, Andrew Tipton <andrew@kiwidrew.com> > wrote: > > Robert: any chance you could share a few more details on the > enhancements > > you're planning for bgworkers? I seem to recall reading that > communicating > > with the dynamic bgworkers after they had been launched was next on > your > > agenda... > > Yeah, it is. I'm working on a patch to allow additional shared memory > segments to be created on the fly. The idea I'm working with is that > a backend that plans to launch a worker will first create a dynamic > shared memory segment, then pass the ID of that segment to the worker > via bgw_main_arg. The worker will map the segment, and then the two > processes can use that to communicate. My thought is to create a > queue abstraction that sits on top of the dynamic shared memory > infrastructure, so that you can set aside a portion of your dynamic > shared memory segment to use as a ring buffer and send messages back > and forth with using some kind of API along these lines: > > extern void dsm_queue_send(dsm_queue *, char *data, uint64 len); > extern uint64 dsm_queue_receive(dsm_queue *, char **dataptr); > > It would also be possible to implement message sending and receiving > using pipes, but I'm leaning away from that because it would require > even more OS-dependent code than I'm already having to write, and > writing OS-dependent shim layers is one of the world's less-rewarding > coding tasks; and also because I think it will be easier to achieve > zero-copy semantics using shared memory. Another idea to get parallel tasks done by bgworkers is rather than dynamically invoking a new bgworker, we can have a set of pre-allocated bgworkers for a sever and then based on need allocate bgworker from pre-allocated array. Now we can allocate shared memory in the beginning based on bgworkers and the information needed to share between Backend and bgworkers (Plan, Tuple, snapshot, .. ). The basic idea can work as below: a. Backend who wishes to get parallel tasks done by bgworker will divide the tasks and check which bgworkers are free and share the plan in corresponding bgworker share memory. b. Bgworker who is polling on its slot of shared memory can retrieve the plan and execute it. c. Bgworker can share the tuples again in its shared memory slot d. Backend can retrieve tuples from shared memory slots of bgworkers where it has communicated the plan e. Backend can send the tuples back to client This idea has a drawback that queue of tuples to be shared has to be of fixed size as we need to allocate memory in beginning. With Regards, Amit Kapila.
On 06/08/13 13:59, Robert Haas wrote: > My thought is to create a > queue abstraction that sits on top of the dynamic shared memory > infrastructure, so that you can set aside a portion of your dynamic > shared memory segment to use as a ring buffer and send messages back > and forth with using some kind of API along these lines: You may find http://quatermass.co.uk/toolsmith/mplib1/ of use here. -- Cheers, Jeremy