Re: TODO : Allow parallel cores to be used by vacuumdb [ WIP ] - Mailing list pgsql-hackers

From Magnus Hagander
Subject Re: TODO : Allow parallel cores to be used by vacuumdb [ WIP ]
Date
Msg-id CABUevEyPFmOtLmqT124BDQtNhHBaDw6YPUijgUXu8UktENMc+A@mail.gmail.com
Whole thread Raw
In response to Re: TODO : Allow parallel cores to be used by vacuumdb [ WIP ]  (Alvaro Herrera <alvherre@2ndquadrant.com>)
Responses Re: TODO : Allow parallel cores to be used by vacuumdb [ WIP ]  (Dilip kumar <dilip.kumar@huawei.com>)
Re: TODO : Allow parallel cores to be used by vacuumdb [ WIP ]  (Dilip kumar <dilip.kumar@huawei.com>)
List pgsql-hackers
<p dir="ltr"><br /> On Jul 16, 2014 7:05 AM, "Alvaro Herrera" <<a
href="mailto:alvherre@2ndquadrant.com">alvherre@2ndquadrant.com</a>>wrote:<br /> ><br /> > Tom Lane wrote:<br
/>> > Dilip kumar <<a href="mailto:dilip.kumar@huawei.com">dilip.kumar@huawei.com</a>> writes:<br /> >
>> On 15 July 2014 19:01, Magnus Hagander Wrote,<br /> > > >> I am late to this game, but the first
thingto my mind was - do we<br /> > > >> really need the whole forking/threading thing on the client at
all?<br/> > ><br /> > > > Thanks for the review, I understand you point, but I think if we have do this
directlyby independent connection,<br /> > > > It's difficult to equally divide the jobs b/w multiple
independentconnections.<br /> > ><br /> > > That argument seems like complete nonsense.  You're confusing
work<br/> > > allocation strategy with the implementation technology for the multiple<br /> > > working
threads. I see no reason why a good allocation strategy couldn't<br /> > > work with either approach; indeed, I
thinkit would likely be easier to<br /> > > do some things *without* client-side physical parallelism, because
that<br/> > > makes it much simpler to handle feedback between the results of different<br /> > >
operationalthreads.<br /> ><br /> > So you would have one initial connection, which generates a task list;<br />
>then open N libpq connections.  Launch one vacuum on each, and then<br /> > sleep on select() on the three
sockets. Whenever one returns<br /> > read-ready, the vacuuming is done and we send another item from the task<br />
>list.  Repeat until tasklist is empty.  No need to fork anything.<br /> ><p dir="ltr">Yeah, those are exactly my
points.I think it would be significantly simpler to do it that way, rather than forking and threading. And also easier
tomake portable... <p dir="ltr">(and as a  optimization on Alvaros suggestion, you can of course reuse the initial
connectionas one of the workers as long as you got the full list of tasks from it up front, which I think you  do
anywayin order to do sorting of tasks...) <p dir="ltr">/Magnus <br /> 

pgsql-hackers by date:

Previous
From: Heikki Linnakangas
Date:
Subject: Re: Bug in spg_range_quad_inner_consistent for adjacent operator (was Re: Add a filed to PageHeaderData)
Next
From: Andres Freund
Date:
Subject: Re: Deadlocks in HS (on 9.0 :( )