Thread: bufmgr code question
In the BufferDesc struct, there seem to be two ways to mark a buffer page as dirty: setting the BM_DIRTY bit mask in the 'flags' field of the struct, and setting the 'cntxDirty' field to true. What is the difference between these two indications of a page's dirtiness? Or, more to the point, is there a reason we have two ways to do what looks like the same thing? BTW, I'd like to remove the behavior that LockBuffer(buf, EXCLUSIVE) automatically marks the page as dirty. Since there are some situations in which we acquire an exclusive buffer lock but don't actually end up modifying the page, this results in dirtying more buffers than are necessary. I think it's also good practise for code that modifies a buffer to explicitly mark it as dirty, rather than depending upon LockBuffer() to do it. Does this sound reasonable, provided I find and catch all the places that depend upon this behavior? (i.e. and change them to explicitly mark the buffer as dirty) -Neil
Neil Conway wrote: > In the BufferDesc struct, there seem to be two ways to mark a buffer > page as dirty: setting the BM_DIRTY bit mask in the 'flags' field of the > struct, and setting the 'cntxDirty' field to true. What is the > difference between these two indications of a page's dirtiness? I don't see any either ... could that be some artefact? > > Or, more to the point, is there a reason we have two ways to do what > looks like the same thing? > > BTW, I'd like to remove the behavior that LockBuffer(buf, EXCLUSIVE) > automatically marks the page as dirty. Since there are some situations > in which we acquire an exclusive buffer lock but don't actually end up > modifying the page, this results in dirtying more buffers than are > necessary. I think it's also good practise for code that modifies a > buffer to explicitly mark it as dirty, rather than depending upon > LockBuffer() to do it. Does this sound reasonable, provided I find and > catch all the places that depend upon this behavior? (i.e. and change > them to explicitly mark the buffer as dirty) Does make sense to me. "Provided you find and catch all" sounds like an "as early as possible in the 7.5 devel cycle" change to me, no? It also looks to me that we are actually modifying the same sourcecode. We should try to avoid conflicting patches. On 10/31 you wrote: > I'd rather see us implement a buffer replacement policy that considers > both frequency + recency (unlike LRU, which considers only recency). > Ideally, that would work "automagically". I'm hoping to get a chance to > implement ARC[1] during the 7.5 cycle. I just posted another fix to my experimental ARC implementation to hackers. From the above I assume you're familiar with the algorithm. Could you please take a look at the "v2" diff and tell me if there's something fundamentally wrong? Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
Neil Conway <neilc@samurai.com> writes: > In the BufferDesc struct, there seem to be two ways to mark a buffer > page as dirty: setting the BM_DIRTY bit mask in the 'flags' field of the > struct, and setting the 'cntxDirty' field to true. What is the > difference between these two indications of a page's dirtiness? > Or, more to the point, is there a reason we have two ways to do what > looks like the same thing? I believe the reason for this is that you are allowed to set cntxDirty to TRUE while holding (only) the context lock on the buffer, while messing with the buffer flags word requires holding (only) the BufMgrLock. Getting rid of cntxDirty would mean additional grabbings of the BufMgrLock when we want to mark buffers dirty. The real solution to this is probably to rethink the rules for locking in the buffer manager. I've thought for some time that the BufMgrLock is a system-wide bottleneck; if we could replace it by finer-grain locks, and in particular use the per-buffer locks for operations affecting just the state of a single buffer, we'd be ahead of the game. > BTW, I'd like to remove the behavior that LockBuffer(buf, EXCLUSIVE) > automatically marks the page as dirty. Yeah. This has been discussed before, see the archives: http://archives.postgresql.org/pgsql-hackers/2002-11/msg00488.php http://archives.postgresql.org/pgsql-hackers/2002-11/msg00679.php http://archives.postgresql.org/pgsql-hackers/2002-11/msg00512.php regards, tom lane
Is there a TODO here? --------------------------------------------------------------------------- Tom Lane wrote: > Neil Conway <neilc@samurai.com> writes: > > In the BufferDesc struct, there seem to be two ways to mark a buffer > > page as dirty: setting the BM_DIRTY bit mask in the 'flags' field of the > > struct, and setting the 'cntxDirty' field to true. What is the > > difference between these two indications of a page's dirtiness? > > Or, more to the point, is there a reason we have two ways to do what > > looks like the same thing? > > I believe the reason for this is that you are allowed to set cntxDirty > to TRUE while holding (only) the context lock on the buffer, while > messing with the buffer flags word requires holding (only) the > BufMgrLock. Getting rid of cntxDirty would mean additional grabbings of > the BufMgrLock when we want to mark buffers dirty. > > The real solution to this is probably to rethink the rules for locking > in the buffer manager. I've thought for some time that the BufMgrLock is > a system-wide bottleneck; if we could replace it by finer-grain locks, > and in particular use the per-buffer locks for operations affecting just > the state of a single buffer, we'd be ahead of the game. > > > BTW, I'd like to remove the behavior that LockBuffer(buf, EXCLUSIVE) > > automatically marks the page as dirty. > > Yeah. This has been discussed before, see the archives: > http://archives.postgresql.org/pgsql-hackers/2002-11/msg00488.php > http://archives.postgresql.org/pgsql-hackers/2002-11/msg00679.php > http://archives.postgresql.org/pgsql-hackers/2002-11/msg00512.php > > regards, tom lane > > ---------------------------(end of broadcast)--------------------------- > TIP 8: explain analyze is your friend > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian <pgman@candle.pha.pa.us> writes: > Is there a TODO here? You needn't add one: I hope to tackle this during the 7.5 dev cycle. -Neil