Thread: Re: [PATCHES] Load Distributed Checkpoints, revised patch
Heikki Linnakangas wrote: > Here's an updated WIP version of the LDC patch. I just spreads the > writes, that achieves the goal of smoothing the checkpoint I/O spikes. I > think sorting the writes etc. is interesting but falls in the category > of further development and should be pushed to 8.4. Why do you think so? Is it too much risk to adapt the sorted writes? The numbers shown by ITAGAKI Takahiro looked quite impressive, at least for large shared_buffers configurations. The reactions where rather positive, too. In general, I am hoping that this patch, together with "Automatic adjustment of bgwriter_lru_maxpages" will finally make default postgresql configurations experience much less impact from check points. For my tast, postgresql has recently got way to many nobs which one must tweak by hand... I welcome any approach on auto-tuning (and auto vacuum!). Patch status says "waiting on update from author": http://archives.postgresql.org/pgsql-patches/2007-04/msg00331.php Any updates on this? Best Regards Michael Paesold
Michael Paesold wrote: > Heikki Linnakangas wrote: >> Here's an updated WIP version of the LDC patch. I just spreads the >> writes, that achieves the goal of smoothing the checkpoint I/O spikes. >> I think sorting the writes etc. is interesting but falls in the >> category of further development and should be pushed to 8.4. > > Why do you think so? Is it too much risk to adapt the sorted writes? The > numbers shown by ITAGAKI Takahiro looked quite impressive, at least for > large shared_buffers configurations. The reactions where rather > positive, too. Well, it is a very recent idea, and it's not clear that it's a win under all circumstances. Adopting that would need more testing, and at this late stage I'd like to just wrap up what we have and come back to this idea for 8.4. If someone performs the tests with different hardware and workloads, I would be willing to consider it; the patch is still at an early stage but it's very isolated change and should therefore be easy to review. But if someone has the hardware and time perform those tests, I'd like them to perform more testing with just LDC first. As Josh pointed out, it would be good to test it with more oddball workloads, all the tests I've done this far have been with DBT-2. > In general, I am hoping that this patch, together with "Automatic > adjustment of bgwriter_lru_maxpages" will finally make default > postgresql configurations experience much less impact from check points. > For my tast, postgresql has recently got way to many nobs which one must > tweak by hand... I welcome any approach on auto-tuning (and auto vacuum!). Sure, but that's another topic. > Patch status says "waiting on update from author": > http://archives.postgresql.org/pgsql-patches/2007-04/msg00331.php > Any updates on this? No. I'm not actually clear what we're waiting for and from whom; I know I haven't had the time to review that in detail yet. IIRC we've seen two very similar patches in the discussions, one from Itagaki-san, and one from Greg Smith. I'm not sure which one of them we should use, they both implement roughly the same thing. But the biggest thing needed for that patch is testing with different workloads. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
On Fri, 2007-06-15 at 13:11 +0200, Michael Paesold wrote: > Heikki Linnakangas wrote: > > Here's an updated WIP version of the LDC patch. I just spreads the > > writes, that achieves the goal of smoothing the checkpoint I/O spikes. I > > think sorting the writes etc. is interesting but falls in the category > > of further development and should be pushed to 8.4. > > Why do you think so? Is it too much risk to adapt the sorted writes? The > numbers shown by ITAGAKI Takahiro looked quite impressive, at least for > large shared_buffers configurations. The reactions where rather > positive, too. Agreed. Seems like a simple isolated piece of code to include or not. If we controlled it with a simple boolean parameter, that would allow testing during beta - I agree it needs more testing. If we find a way of automating it, cool. If its flaky, strip it out before we release. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas <heikki@enterprisedb.com> wrote: > Michael Paesold wrote: > > Why do you think so? Is it too much risk to adapt the sorted writes? The > > numbers shown by ITAGAKI Takahiro looked quite impressive, at least for > > large shared_buffers configurations. The reactions where rather > > positive, too. > > Well, it is a very recent idea, and it's not clear that it's a win under > all circumstances. Adopting that would need more testing, and at this > late stage I'd like to just wrap up what we have and come back to this > idea for 8.4. I'm planning it for 8.4. Through the discussion of LDC and the sorted writes, we found that kernels are not so clever as we don't need to schedule I/O. At least, we'd better to give them more hints. Many self I/O management methods have been often proposed; read-ahead, aio, bulk extension of files and so on. The sorted writes is one of them. We should test them in various workloads and take what is good. Regards, --- ITAGAKI Takahiro NTT Open Source Software Center