Thread: Win32 and fsync()
For Win32, in order to emulate fsync() we will need to call FlushFileBuffers(): http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/ base/flushfilebuffers.asp
I'm having difficulty digging up the reference, but I think I recall seeing something that said, roughly, on W32 there are 2 sets of buffers - those in the user level library and those in the kernel level driver, and FlushFileBuffers drains the first, while _commit drains both (it includes a call to FlushFileBuffers). I'm also fairly sure I saw something like #define fsync _commit in the Berkeley DB sources the other day, which might be a clue. I'll be happy to be corrected, though. cheers andrew ----- Original Message ----- From: "Dann Corbit" <DCorbit@connx.com> To: "PostgreSQL Hackers" <pgsql-hackers@postgresql.org> Sent: Monday, February 03, 2003 3:15 PM Subject: [HACKERS] Win32 and fsync() > For Win32, in order to emulate fsync() we will need to call > FlushFileBuffers(): > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/ > base/flushfilebuffers.asp > > ---------------------------(end of broadcast)--------------------------- > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>> For Win32, in order to emulate fsync() we will need to call >> FlushFileBuffers(): The supplied link does not work. FlushFileBuffers() is for flushing files opened with CreateFile() etc. For files opened with fopen(), call fflush(). For files opened with _open(), call _commit(). Likekly the win32 port code, which I have yet to inspect, will use the second version. Merlin
> -----Original Message----- > From: Andrew Dunstan [mailto:andrew@dunslane.net] > Sent: 03 February 2003 22:47 > To: PostgreSQL Hackers > Subject: Re: [HACKERS] Win32 and fsync() > > > I'm having difficulty digging up the reference, but I think I > recall seeing something that said, roughly, on W32 there are > 2 sets of buffers - those in the user level library and those > in the kernel level driver, and FlushFileBuffers drains the > first, while _commit drains both (it includes a call to > FlushFileBuffers). > > I'm also fairly sure I saw something like > #define fsync _commit > in the Berkeley DB sources the other day, which might be a clue. > > I'll be happy to be corrected, though. I too have yet to look at the Win32 patces, but if they have used FlushFileBuffers, perhaps that would explain my powerfail test results... Regards, Dave.
>I'm having difficulty digging up the reference, but I think I recall
seeing >something that said, roughly, on W32 there are 2 sets of buffers
- those in >the user level library and those in the kernel level driver,
and >FlushFileBuffers drains the first, while _commit drains both (it
includes a >call to FlushFileBuffers).
You were correct: here is the source.
int __cdecl _commit ( int filedes )
{ int retval;
/* if filedes out of range, complain */ if ( ((unsigned)filedes >= (unsigned)_nhandle) ||
!(_osfile(filedes)& FOPEN) ) { errno = EBADF; return (-1); }
_lock_fh(filedes);
/* if filedes open, try to commit, else fall through to bad */ if (_osfile(filedes) & FOPEN) {
if ( !FlushFileBuffers((HANDLE)_get_osfhandle(filedes))
) { retval = GetLastError(); } else { retval = 0; /*
returnsuccess */ }
/* map the OS return code to C errno value and return
code */ if (retval == 0) { goto good; } else {
_doserrno= retval; goto bad; }
}
bad : errno = EBADF; retval = -1;
good : _unlock_fh(filedes); return (retval);
}
On Mon, 3 Feb 2003, Andrew Dunstan wrote: > I'm also fairly sure I saw something like > #define fsync _commit > in the Berkeley DB sources the other day, which might be a clue. > > I'll be happy to be corrected, though. You'd be right: /** Win32 has fsync, getcwd, snprintf and vsnprintf, but under different names.*/ #define fsync(fd) _commit(fd) Funny Win32-ism Still not sure if it has the same semantics as Unix fsync() but sleepycat (Berkeley) DB seems to think so. Gavin
> -----Original Message-----
> From: Merlin Moncure [mailto:merlin.moncure@rcsonline.com]
> Sent: Monday, February 03, 2003 3:00 PM
> To: Andrew Dunstan
> Cc: pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] Win32 and fsync()
>
>
> >I'm having difficulty digging up the reference, but I think I recall
> seeing >something that said, roughly, on W32 there are 2 sets
> of buffers
> - those in >the user level library and those in the kernel
> level driver, and >FlushFileBuffers drains the first, while
> _commit drains both (it includes a >call to FlushFileBuffers).
>
> You were correct: here is the source.
>
>
> int __cdecl _commit (
> int filedes
> )
> {
> int retval;
>
> /* if filedes out of range, complain */
> if ( ((unsigned)filedes >= (unsigned)_nhandle) ||
> !(_osfile(filedes) & FOPEN) )
> {
> errno = EBADF;
> return (-1);
> }
>
> _lock_fh(filedes);
>
> /* if filedes open, try to commit, else fall through to bad */
> if (_osfile(filedes) & FOPEN) {
>
> if (
> !FlushFileBuffers((HANDLE)_get_osfhandle(filedes))
> ) {
> retval = GetLastError();
> } else {
> retval = 0; /* return success */
> }
>
> /* map the OS return code to C errno value
> and return code */
> if (retval == 0) {
> goto good;
> } else {
> _doserrno = retval;
> goto bad;
> }
>
> }
>
> bad :
> errno = EBADF;
> retval = -1;
> good :
> _unlock_fh(filedes);
> return (retval);
> }
Where is the "other" flush besides FlushFileBuffers()?
On Mon, 3 Feb 2003, Dann Corbit wrote:
> > -----Original Message-----
> > From: Merlin Moncure [mailto:merlin.moncure@rcsonline.com]
> > Sent: Monday, February 03, 2003 3:00 PM
> > To: Andrew Dunstan
> > Cc: pgsql-hackers@postgresql.org
> > Subject: Re: [HACKERS] Win32 and fsync()
> >
> >
> > >I'm having difficulty digging up the reference, but I think I recall
> > seeing >something that said, roughly, on W32 there are 2 sets
> > of buffers
> > - those in >the user level library and those in the kernel
> > level driver, and >FlushFileBuffers drains the first, while
> > _commit drains both (it includes a >call to FlushFileBuffers).
> >
> > You were correct: here is the source.
> >
> >
> > int __cdecl _commit (
> > int filedes
> > )
> > {
> > int retval;
> >
> > /* if filedes out of range, complain */
> > if ( ((unsigned)filedes >= (unsigned)_nhandle) ||
> > !(_osfile(filedes) & FOPEN) )
> > {
> > errno = EBADF;
> > return (-1);
> > }
> >
> > _lock_fh(filedes);
> >
> > /* if filedes open, try to commit, else fall through to bad */
> > if (_osfile(filedes) & FOPEN) {
> >
> > if (
> > !FlushFileBuffers((HANDLE)_get_osfhandle(filedes))
> > ) {
> > retval = GetLastError();
> > } else {
> > retval = 0; /* return success */
> > }
> >
> > /* map the OS return code to C errno value
> > and return code */
> > if (retval == 0) {
> > goto good;
> > } else {
> > _doserrno = retval;
> > goto bad;
> > }
> >
> > }
> >
> > bad :
> > errno = EBADF;
> > retval = -1;
> > good :
> > _unlock_fh(filedes);
> > return (retval);
> > }
>
> Where is the "other" flush besides FlushFileBuffers()?
The only real code there is, it seems, an exclusive look on the file
descriptor. (Provided of course that that is what _lock_fh(filedes) does).
Gavin
I'm not sure what the provenance of this code for _commit is - I thought it
was in the standard MS libs - have they finally seen the light and open
sourced it? ;-)
andrew
----- Original Message -----
From: "Dann Corbit" <DCorbit@connx.com>
To: "Merlin Moncure" <merlin.moncure@rcsonline.com>; "Andrew Dunstan"
<andrew@dunslane.net>
Cc: <pgsql-hackers@postgresql.org>
Sent: Monday, February 03, 2003 6:24 PM
Subject: Re: [HACKERS] Win32 and fsync()
> > -----Original Message-----
> > From: Merlin Moncure [mailto:merlin.moncure@rcsonline.com]
> > Sent: Monday, February 03, 2003 3:00 PM
> > To: Andrew Dunstan
> > Cc: pgsql-hackers@postgresql.org
> > Subject: Re: [HACKERS] Win32 and fsync()
> >
> >
> > >I'm having difficulty digging up the reference, but I think I recall
> > seeing >something that said, roughly, on W32 there are 2 sets
> > of buffers
> > - those in >the user level library and those in the kernel
> > level driver, and >FlushFileBuffers drains the first, while
> > _commit drains both (it includes a >call to FlushFileBuffers).
> >
> > You were correct: here is the source.
> >
> >
> > int __cdecl _commit (
> > int filedes
> > )
> > {
> > int retval;
> >
> > /* if filedes out of range, complain */
> > if ( ((unsigned)filedes >= (unsigned)_nhandle) ||
> > !(_osfile(filedes) & FOPEN) )
> > {
> > errno = EBADF;
> > return (-1);
> > }
> >
> > _lock_fh(filedes);
> >
> > /* if filedes open, try to commit, else fall through to bad */
> > if (_osfile(filedes) & FOPEN) {
> >
> > if (
> > !FlushFileBuffers((HANDLE)_get_osfhandle(filedes))
> > ) {
> > retval = GetLastError();
> > } else {
> > retval = 0; /* return success */
> > }
> >
> > /* map the OS return code to C errno value
> > and return code */
> > if (retval == 0) {
> > goto good;
> > } else {
> > _doserrno = retval;
> > goto bad;
> > }
> >
> > }
> >
> > bad :
> > errno = EBADF;
> > retval = -1;
> > good :
> > _unlock_fh(filedes);
> > return (retval);
> > }
>
> Where is the "other" flush besides FlushFileBuffers()?
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> -----Original Message----- > From: Andrew Dunstan [mailto:andrew@dunslane.net] > Sent: Monday, February 03, 2003 3:31 PM > To: Dann Corbit; Merlin Moncure > Cc: pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] Win32 and fsync() > > > I'm not sure what the provenance of this code for _commit is > - I thought it was in the standard MS libs - have they > finally seen the light and open sourced it? ;-) MS has always shipped the source code to their standard libraries on the CD. However, the compiler source remains closed. This seems insignificant, but sometimes functions are totally inlined within the compiler (e.g. math functions like sin() and exp()).
----- Original Message ----- From: "Gavin Sherry" <swm@linuxworld.com.au> [snip] > > Where is the "other" flush besides FlushFileBuffers()? > > The only real code there is, it seems, an exclusive look on the file > descriptor. (Provided of course that that is what _lock_fh(filedes) does). > yes, it looks that way. I wonder what whould happen to reliability and performance if the files were opened using Windows' write through cache flag (FILE_FLAG_WRITE_THROUGH). Also, this article might be relevant: http://support.microsoft.com/default.aspx?scid=kb;en-us;332023 although I see the Windows tests were done on W2000/SP3, so maybe not. andrew
I think the fopen or _open family of functions all map directly to the win32 API. They add a little cruft, which generally makes using them pointless, because you have less control over security, caching, and other such things when opening the file. There is the slight overhead of the extra call, and the conversion of the C handle to the win32 handle. However, all you get with the api is readfile() and writefile(), which means workhorses like fprintf() and especially fgets() and must be reimplemented (in lieu of using the C++ runtime). > MS has always shipped the source code to their standard libraries on the > CD. However, the compiler source remains closed. This seems > insignificant, but sometimes functions are totally inlined within the > compiler (e.g. math functions like sin() and exp()). They have a special calling convention __dllspec(naked), which strips the function prolog and epilog and allows you to write your own in assembly. You can use that to write very fast functions when you don't trust the optimizer to inline your functions. I think your are correct some of the math functions are written this way. Merlin