Thread: Background writer committed
I committed the first part of the background writer process. We had a consensus on attempting to avoid write() calls from regular backends, but did no come to any conclusions what to do to force the kernel to actually do some IO. Consequently, this patch is a separate process launched by postmaster, that periodically write()'s out "some" dirty buffers in LRU order. This causes the buffers returned for replacement (when a backend needs to read in a page) to be clean allways. The process does no sync(), fsync() or any other calls thus far. Nothing has changed in the checkpoint logic either. The configuration options controlling the process are all PGC_SIGHUP: # - Background writer - #bgwriter_delay = 200 # 10-5000 milliseconds #bgwriter_percent = 1 # 0-100% of dirty buffers #bgwriter_maxpages = 100 # 1-1000 buffers max at once Delay is the number of milliseconds to wait between loops. If there was nothing to do at all in one loop (all buffers clean), then the process will sleep for 10 seconds. Percent is the percentage of "dirty pages" to write per loop. This is independant of the size of the buffer pool. If percent = 0 the postmaster will not start the process at all. Maxpages is an upper bound to prevent the background writer from producing a write storm if a sequential operation causes all pages of a large buffer pool to be dirtied at once. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
TODO updated: * -Use background process to write dirty shared buffers to disk --------------------------------------------------------------------------- Jan Wieck wrote: > I committed the first part of the background writer process. We had a > consensus on attempting to avoid write() calls from regular backends, > but did no come to any conclusions what to do to force the kernel to > actually do some IO. > > Consequently, this patch is a separate process launched by postmaster, > that periodically write()'s out "some" dirty buffers in LRU order. This > causes the buffers returned for replacement (when a backend needs to > read in a page) to be clean allways. The process does no sync(), fsync() > or any other calls thus far. Nothing has changed in the checkpoint logic > either. > > The configuration options controlling the process are all PGC_SIGHUP: > > # - Background writer - > #bgwriter_delay = 200 # 10-5000 milliseconds > #bgwriter_percent = 1 # 0-100% of dirty buffers > #bgwriter_maxpages = 100 # 1-1000 buffers max at once > > Delay is the number of milliseconds to wait between loops. If there was > nothing to do at all in one loop (all buffers clean), then the process > will sleep for 10 seconds. > > Percent is the percentage of "dirty pages" to write per loop. This is > independant of the size of the buffer pool. If percent = 0 the > postmaster will not start the process at all. > > Maxpages is an upper bound to prevent the background writer from > producing a write storm if a sequential operation causes all pages of a > large buffer pool to be dirtied at once. > > > Jan > > -- > #======================================================================# > # It's easier to get forgiveness for being wrong than for being right. # > # Let's break this rule - forgive me. # > #================================================== JanWieck@Yahoo.com # > > > ---------------------------(end of broadcast)--------------------------- > TIP 8: explain analyze is your friend > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Jan Wieck wrote: > I committed the first part of the background writer process. We had a > consensus on attempting to avoid write() calls from regular backends, > but did no come to any conclusions what to do to force the kernel to > actually do some IO. > > Consequently, this patch is a separate process launched by postmaster, > that periodically write()'s out "some" dirty buffers in LRU order. This > causes the buffers returned for replacement (when a backend needs to > read in a page) to be clean allways. The process does no sync(), fsync() > or any other calls thus far. Nothing has changed in the checkpoint logic > either. Can we have some idea where to tweak sync routines for comparing results? I mean I would like to run pgbench with same config all along and compare the performance difference between sync, fsync and fdatasync etc. If we could get to run any live world data test by that, it would be great as well. Shridhar
Shridhar Daithankar wrote: > Jan Wieck wrote: > >> I committed the first part of the background writer process. We had a >> consensus on attempting to avoid write() calls from regular backends, >> but did no come to any conclusions what to do to force the kernel to >> actually do some IO. >> >> Consequently, this patch is a separate process launched by postmaster, >> that periodically write()'s out "some" dirty buffers in LRU order. This >> causes the buffers returned for replacement (when a backend needs to >> read in a page) to be clean allways. The process does no sync(), fsync() >> or any other calls thus far. Nothing has changed in the checkpoint logic >> either. > > Can we have some idea where to tweak sync routines for comparing results? > > I mean I would like to run pgbench with same config all along and compare the > performance difference between sync, fsync and fdatasync etc. pgbench is actually a very bad example to test any cache strategy. Either 98% of your lookups result in cache hits, so basically your entire database is cached, or it doesn't fit and every cache strategy becomes useless. It doesn't have parts that fit and other parts that don't. I think pgbench doesn't use non-uniform random access as real world applications do (you have bestsellers and other items, you have frequent customers and once-a-year visitors). So it's very hard to get the system into a state where you have like 50% cache hitrate. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #