Thread: fix use of posix_fadvise in xlog.c
Hi all, I wanted to propose a fix for to xlog.c regarding the use of posix_fadvise() for 9.1 (unless someone feels it's ok for 9.0). Currently posix_fadvise() is used right before a log file is closed so it's effectively not doing anything, when posix_fadvise is to be called. This patch moves the posix_fadvise() call into 3 other locations within XLogFileInit() where a file handle is returned. The first case is where an existing open file handle is returned. The next case is when a file is to be zeroed out. The third case is returning a file handle, which may be the file that was just zeroed out. Does this look ok? Regards, Mark
Attachment
On 10/06/10 06:47, Mark Wong wrote: > I wanted to propose a fix for to xlog.c regarding the use of > posix_fadvise() for 9.1 (unless someone feels it's ok for 9.0). > Currently posix_fadvise() is used right before a log file is closed so > it's effectively not doing anything, when posix_fadvise is to be > called. This patch moves the posix_fadvise() call into 3 other > locations within XLogFileInit() where a file handle is returned. The > first case is where an existing open file handle is returned. The > next case is when a file is to be zeroed out. The third case is > returning a file handle, which may be the file that was just zeroed > out. I don't think POSIX_FADV_DONTNEED does what you think it does. It tells the kernel that "you don't need to keep these pages in the cache anymore, I won't be accessing them anymore". If you call it when you open the file, before reading/writing, there is nothing in the cache and the call will do nothing. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
On Jun 9, 2010, at 11:25 PM, Heikki Linnakangas <heikki.linnakangas@enterprisedb.com > wrote: > On 10/06/10 06:47, Mark Wong wrote: >> I wanted to propose a fix for to xlog.c regarding the use of >> posix_fadvise() for 9.1 (unless someone feels it's ok for 9.0). >> Currently posix_fadvise() is used right before a log file is closed >> so >> it's effectively not doing anything, when posix_fadvise is to be >> called. This patch moves the posix_fadvise() call into 3 other >> locations within XLogFileInit() where a file handle is returned. The >> first case is where an existing open file handle is returned. The >> next case is when a file is to be zeroed out. The third case is >> returning a file handle, which may be the file that was just zeroed >> out. > > I don't think POSIX_FADV_DONTNEED does what you think it does. It > tells the kernel that "you don't need to keep these pages in the > cache anymore, I won't be accessing them anymore". If you call it > when you open the file, before reading/writing, there is nothing in > the cache and the call will do nothing. Oops, my bad. I think I was confused by the short description in the man page. I didn't read the longer descriptoon. :( Then would it be worth making the this call after the file is zeroed out? Regards, Mark
On 10/06/10 18:17, Mark Wong wrote: > On Jun 9, 2010, at 11:25 PM, Heikki Linnakangas > <heikki.linnakangas@enterprisedb.com> wrote: >> I don't think POSIX_FADV_DONTNEED does what you think it does. It >> tells the kernel that "you don't need to keep these pages in the cache >> anymore, I won't be accessing them anymore". If you call it when you >> open the file, before reading/writing, there is nothing in the cache >> and the call will do nothing. > > Oops, my bad. I think I was confused by the short description in the man > page. I didn't read the longer descriptoon. :( Then would it be worth > making the this call after the file is zeroed out? Not sure. If you're churning through WAL files at a reasonable speed, the zeroed-out file will soon be written to again. OTOH, we always write whole pages, so maybe the OS is smart enough to not read the page back to memory just to overwrite it. In a steady-state situation new WAL files are not created very often because we recycle old ones, so it probably doesn't make much difference. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes:
> In a steady-state situation new WAL files are not created very often
> because we recycle old ones, so it probably doesn't make much difference.
Yeah. We really don't worry too much about the performance of the
new-WAL-file-creation code path because of this.
regards, tom lane
Tom Lane wrote: > Heikki Linnakangas <heikki.linnakangas@enterprisedb.com> writes: > >> In a steady-state situation new WAL files are not created very often >> because we recycle old ones, so it probably doesn't make much difference. >> > > Yeah. We really don't worry too much about the performance of the > new-WAL-file-creation code path because of this. > The only situation where the WAL zeroing path turns ugly is if you launch a bunch of activity against a fresh server that doesn't have any segments to recycle yet. The last time we talked about improving that, the best idea I thought came out was to be better about preallocating segments than the code already is, rather than trying to speed up how the kernel deals with the situation. See the links for "Be more aggressive about creating WAL files" at http://wiki.postgresql.org/wiki/Todo I'm also not very optimistic about adding more posix_fadvise calls really helping just because the implementations of those are so unpredictable across operating systems. I'm sure that Mark could figure out the right magic to speed up this specific case on Linux, but have my doubts that work would translate very well to many other operating systems. Whereas a more generic preallocation improvement would help everywhere. -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.us