Thread: Win32 rename()/unlink() questions
I am working with several groups getting the Win32 port ready for 7.4 and I have a few questions: What is the standard workaround for the fact that rename() isn't atomic on Win32? Do we need to create our own locking around the reading/writing of files that are normally updated in place using rename()? Second, when you unlink() a file on Win32, do applications continue accessing the old file contents if they had the file open before the unlink? -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Wed, Sep 18, 2002 at 08:01:42PM -0400, Bruce Momjian wrote: > Second, when you unlink() a file on Win32, do applications continue > accessing the old file contents if they had the file open before the > unlink? I'm pretty sure it errors with 'file in use'. Pretty ugly, huh? Ross
> On Wed, Sep 18, 2002 at 08:01:42PM -0400, Bruce Momjian wrote: > > > Second, when you unlink() a file on Win32, do applications continue > > accessing the old file contents if they had the file open before the > > unlink? > > I'm pretty sure it errors with 'file in use'. Pretty ugly, huh? Yeah - the windows filesystem is pretty poor when it comes to multiuser access. That's why even as administrator I cannot delete borked files and people's profiles and stuff off our NT server - the files are always 'in use'. Even if you kick all users off, reboot the machine, do whatever. It's terrible. Chris
Bruce Momjian wrote: > I am working with several groups getting the Win32 port ready for 7.4 > and I have a few questions: > > What is the standard workaround for the fact that rename() isn't atomic > on Win32? Do we need to create our own locking around the > reading/writing of files that are normally updated in place using > rename()? Visual C++ comes with the source to Microsoft's C library: rename() calls MoveFile() which will error if: 1. The target file exists 2. The source file is in use MoveFileEx() (not available on 95/98) can overwrite the target file if it exists. The Apache APR portability library uses MoveFileEx() to rename files if under NT/XP/2K vs. a sequence of : 1. CreateFile() to test for target file existence 2. DeleteFile() to remove the target file 3. MoveFile() to rename the old file to new under Windows 95/98. Of course, some other process could create the target file between 2 and 3, so their rename() would just error out in that situation. I haven't tested it, but I recall reading somewhere that MoveFileEx() has the ability to rename an opened file. I'm 99% sure MoveFile() will fail if the source file is open. > > Second, when you unlink() a file on Win32, do applications continue > accessing the old file contents if they had the file open before the > unlink? > unlink() just calls DeleteFile() which will error if: 1. The target file is in use CreateFile() has the option: FILE_FLAG_DELETE_ON_CLOSE which might be able to be used to simulate traditional unlink() behavior. Hope that helps, Mike Mascari mascarm@mascari.com
Christopher Kings-Lynne wrote: >>On Wed, Sep 18, 2002 at 08:01:42PM -0400, Bruce Momjian wrote: >> >> >>>Second, when you unlink() a file on Win32, do applications continue >>>accessing the old file contents if they had the file open before the >>>unlink? >> >>I'm pretty sure it errors with 'file in use'. Pretty ugly, huh? > > > Yeah - the windows filesystem is pretty poor when it comes to multiuser > access. That's why even as administrator I cannot delete borked files and > people's profiles and stuff off our NT server - the files are always 'in > use'. Even if you kick all users off, reboot the machine, do whatever. > It's terrible.>> Chris> Yep. That's why often it requires rebooting to uninstall software. How can the installer remove itself? Under Windows 95/98/ME, you have to manually add entries to WININIT.INI. With Windows NT/XP/2K, MoveFileEx() with a NULL target and the MOVEFILE_DELAY_UNTIL_REBOOT flag will add the appropriate entries into the system registry so that the next time the machine reboots it will remove the files specified. Its a real pain and a real hack of an OS. Mike Mascari mascarm@mascari.com
Mike Mascari wrote:
> Bruce Momjian wrote:
> > I am working with several groups getting the Win32 port ready for 7.4
> > and I have a few questions:
> >
> > What is the standard workaround for the fact that rename() isn't atomic
> > on Win32? Do we need to create our own locking around the
> > reading/writing of files that are normally updated in place using
> > rename()?
>
> Visual C++ comes with the source to Microsoft's C library:
>
> rename() calls MoveFile() which will error if:
>
> 1. The target file exists
> 2. The source file is in use
>
> MoveFileEx() (not available on 95/98) can overwrite the target
> file if it exists. The Apache APR portability library uses
> MoveFileEx() to rename files if under NT/XP/2K vs. a sequence of :
>
> 1. CreateFile() to test for target file existence
> 2. DeleteFile() to remove the target file
> 3. MoveFile() to rename the old file to new
>
> under Windows 95/98. Of course, some other process could create
> the target file between 2 and 3, so their rename() would just
> error out in that situation. I haven't tested it, but I recall
> reading somewhere that MoveFileEx() has the ability to rename an
> opened file. I'm 99% sure MoveFile() will fail if the source
> file is open.
OK, I downloaded APR and see in apr_file_rename():
if (MoveFileEx(frompath, topath, MOVEFILE_REPLACE_EXISTING |
MOVEFILE_COPY_ALLOWED))
Looking at the entire APR function, they have lots of tests so it works
on Win9X and wide characters. I think we will just use the APR as a
guide in implementing the things we need. I think MoveFileEx() is the
proper way to go; any other solution requires loop tests for rename.
I see the MoveFileEx manual page at:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/movefile.asp
> > Second, when you unlink() a file on Win32, do applications continue
> > accessing the old file contents if they had the file open before the
> > unlink?
> >
>
> unlink() just calls DeleteFile() which will error if:
>
> 1. The target file is in use
>
> CreateFile() has the option:
>
> FILE_FLAG_DELETE_ON_CLOSE
>
> which might be able to be used to simulate traditional unlink()
> behavior.
No, that flag isn't going to help us. I wonder what MoveFileEx does if
the target file exists _and_ is open by another user? I don't see any
loop in that Win32 rename() routine, and I looked at the Unix version of
apr_file_rename and its just a straight rename() call.
-- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610)
359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square,
Pennsylvania19073
Bruce Momjian wrote: > > > Second, when you unlink() a file on Win32, do applications continue > > > accessing the old file contents if they had the file open before the > > > unlink? > > > > > > > unlink() just calls DeleteFile() which will error if: > > > > 1. The target file is in use > > > > CreateFile() has the option: > > > > FILE_FLAG_DELETE_ON_CLOSE > > > > which might be able to be used to simulate traditional unlink() > > behavior. > > No, that flag isn't going to help us. I wonder what MoveFileEx does if > the target file exists _and_ is open by another user? I don't see any > loop in that Win32 rename() routine, and I looked at the Unix version of > apr_file_rename and its just a straight rename() call. This says that if the target is in use, it is overwritten: http://support.microsoft.com/default.aspx?scid=KB;EN-US;q140570& While I think that is good news, does it open the problem of other readers reading partial updates to the file and therefore seeing garbage. Not sure how to handle that, nor am I even sure how I would test it. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian wrote: > Bruce Momjian wrote: >>> >>>unlink() just calls DeleteFile() which will error if: >>> >>>1. The target file is in use >>> >>>CreateFile() has the option: >>> >>>FILE_FLAG_DELETE_ON_CLOSE >>> >>>which might be able to be used to simulate traditional unlink() >>>behavior. >> >>No, that flag isn't going to help us. I wonder what MoveFileEx does if >>the target file exists _and_ is open by another user? I don't see any >>loop in that Win32 rename() routine, and I looked at the Unix version of >>apr_file_rename and its just a straight rename() call. > > > This says that if the target is in use, it is overwritten: > > http://support.microsoft.com/default.aspx?scid=KB;EN-US;q140570& I read the article and did not come away with that conclusion. The article describes using the MOVEFILE_DELAY_UNTIL_REBOOT flag, which was created for the express purpose of allowing a SETUP.EXE to remove itself, or rather tell Windows to remove it on the next reboot. Also, if you want the Win32 port to run in 95/98/ME, you can't rely on MoveFileEx(), you have to use MoveFile(). I will do some testing with concurrency and let you know. But don't get your hopes up. This is one of the many advantages that TABLESPACEs have when more than one relation is stored in a single DATAFILE. There was Oracle for MS-DOS, after all.. Mike Mascari mascarm@mascari.com
Mike Mascari wrote: > I read the article and did not come away with that conclusion. > The article describes using the MOVEFILE_DELAY_UNTIL_REBOOT > flag, which was created for the express purpose of allowing a > SETUP.EXE to remove itself, or rather tell Windows to remove it > on the next reboot. Also, if you want the Win32 port to run in > 95/98/ME, you can't rely on MoveFileEx(), you have to use > MoveFile(). > > I will do some testing with concurrency and let you know. But > don't get your hopes up. This is one of the many advantages that > TABLESPACEs have when more than one relation is stored in a > single DATAFILE. There was Oracle for MS-DOS, after all.. I was focusing on handling of pg_pwd and other config file that are written by various backend while other backends are reading them. The actual data files should be OK because we have an exclusive lock when we are adding/removing them. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian wrote: > Mike Mascari wrote: > >>I will do some testing with concurrency and let you know. But >>don't get your hopes up. This is one of the many advantages that >>TABLESPACEs have when more than one relation is stored in a >>single DATAFILE. There was Oracle for MS-DOS, after all.. > > > I was focusing on handling of pg_pwd and other config file that are > written by various backend while other backends are reading them. The > actual data files should be OK because we have an exclusive lock when we > are adding/removing them. > OK. So you want to test: 1. Process 1 opens "foo" 2. Process 2 opens "foo" 3. Process 1 renames "foo" to "bar" 4. Process 2 can safely read from its open file handle Is that what you want tested? I have a small Win32 app ready to test. Just let me know the scenarios... Mike Mascari mascarm@mascari.com
Mike Mascari wrote: > Bruce Momjian wrote: > >> Mike Mascari wrote: >> >>> I will do some testing with concurrency and let you know. But don't >>> get your hopes up. This is one of the many advantages that >>> TABLESPACEs have when more than one relation is stored in a single >>> DATAFILE. There was Oracle for MS-DOS, after all.. >> >> >> >> I was focusing on handling of pg_pwd and other config file that are >> written by various backend while other backends are reading them. The >> actual data files should be OK because we have an exclusive lock when we >> are adding/removing them. >> > > OK. So you want to test: > > 1. Process 1 opens "foo" > 2. Process 2 opens "foo" > 3. Process 1 renames "foo" to "bar" > 4. Process 2 can safely read from its open file handle Actually, looking at the pg_pwd code, you want to determine a way for: 1. Process 1 opens "foo" 2. Process 2 opens "foo" 3. Process 1 creates "bar" 4. Process 1 renames "bar" to "foo" 5. Process 2 can continue to read data from the open file handle and get the original "foo" data. Is that correct? Mike Mascari mascarm@mascari.com
Mike Mascari wrote: > Actually, looking at the pg_pwd code, you want to determine a > way for: > > 1. Process 1 opens "foo" > 2. Process 2 opens "foo" > 3. Process 1 creates "bar" > 4. Process 1 renames "bar" to "foo" > 5. Process 2 can continue to read data from the open file handle > and get the original "foo" data. Yep, that's it. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian wrote:
> Mike Mascari wrote:
>
>>Actually, looking at the pg_pwd code, you want to determine a
>>way for:
>>
>>1. Process 1 opens "foo"
>>2. Process 2 opens "foo"
>>3. Process 1 creates "bar"
>>4. Process 1 renames "bar" to "foo"
>>5. Process 2 can continue to read data from the open file handle
>>and get the original "foo" data.
>
>
> Yep, that's it.
>
So far, MoveFileEx("foo", "bar", MOVEFILE_REPLACE_EXISTING)
returns "Access Denied" when Process 1 attempts the rename. But
I'm continuing to investigate the possibilities...
Mike Mascari
mascarm@mascari.com
On Fri, 20 Sep 2002, Mike Mascari wrote:
> Bruce Momjian wrote:
> > Mike Mascari wrote:
> >
> >>Actually, looking at the pg_pwd code, you want to determine a
> >>way for:
> >>
> >>1. Process 1 opens "foo"
> >>2. Process 2 opens "foo"
> >>3. Process 1 creates "bar"
> >>4. Process 1 renames "bar" to "foo"
> >>5. Process 2 can continue to read data from the open file handle
> >>and get the original "foo" data.
> >
> >
> > Yep, that's it.
> >
>
> So far, MoveFileEx("foo", "bar", MOVEFILE_REPLACE_EXISTING)
> returns "Access Denied" when Process 1 attempts the rename. But
> I'm continuing to investigate the possibilities...
Does a sequence like
Process 1 opens "foo"
Process 2 opens "foo"
Process 1 creates "bar"
Process 1 renames "foo" to <something>- where something is generated to not overlap an existing file
Process 1 renames "bar" to "foo"
Process 2 continues reading
let you do the replace and keep reading (at the penalty that
you've now got to have a way to know when to remove the
various <something>s)
Stephan Szabo wrote:
> On Fri, 20 Sep 2002, Mike Mascari wrote:
>>Bruce Momjian wrote:
>>>Mike Mascari wrote:
>>>>Actually, looking at the pg_pwd code, you want to determine a
>>>>way for:
>>>>
>>>>1. Process 1 opens "foo"
>>>>2. Process 2 opens "foo"
>>>>3. Process 1 creates "bar"
>>>>4. Process 1 renames "bar" to "foo"
>>>>5. Process 2 can continue to read data from the open file handle
>>>>and get the original "foo" data.
>>>
>>>
>>>Yep, that's it.
>>>
>>
>>So far, MoveFileEx("foo", "bar", MOVEFILE_REPLACE_EXISTING)
>>returns "Access Denied" when Process 1 attempts the rename. But
>>I'm continuing to investigate the possibilities...
>
>
> Does a sequence like
> Process 1 opens "foo"
> Process 2 opens "foo"
> Process 1 creates "bar"
> Process 1 renames "foo" to <something>
> - where something is generated to not overlap an existing file
> Process 1 renames "bar" to "foo"
> Process 2 continues reading
> let you do the replace and keep reading (at the penalty that
> you've now got to have a way to know when to remove the
> various <something>s)
Yes! Indeed that does work.
Mike Mascari
mascarm@mascari.com
On Fri, 20 Sep 2002, Mike Mascari wrote:
> Stephan Szabo wrote:
> > On Fri, 20 Sep 2002, Mike Mascari wrote:
> >>So far, MoveFileEx("foo", "bar", MOVEFILE_REPLACE_EXISTING)
> >>returns "Access Denied" when Process 1 attempts the rename. But
> >>I'm continuing to investigate the possibilities...
> >
> >
> > Does a sequence like
> > Process 1 opens "foo"
> > Process 2 opens "foo"
> > Process 1 creates "bar"
> > Process 1 renames "foo" to <something>
> > - where something is generated to not overlap an existing file
> > Process 1 renames "bar" to "foo"
> > Process 2 continues reading
> > let you do the replace and keep reading (at the penalty that
> > you've now got to have a way to know when to remove the
> > various <something>s)
>
> Yes! Indeed that does work.
Thinking back, I think that may still fail on Win95 (using MoveFile).
Once in the past I had to work on (un)installers for Win* and I
vaguely remember Win95 being more strict than Win98 but that may just
have been with moving the executable you're currently running.
Stephan Szabo wrote: > On Fri, 20 Sep 2002, Mike Mascari wrote: >> >>Yes! Indeed that does work. > > > Thinking back, I think that may still fail on Win95 (using MoveFile). > Once in the past I had to work on (un)installers for Win* and I > vaguely remember Win95 being more strict than Win98 but that may just > have been with moving the executable you're currently running. Well, here's the test: foo.txt contains "This is FOO!" bar.txt contains "This is BAR!" Process 1 opens foo.txt Process 2 opens foo.txt Process 1 sleeps 7.5 seconds Process 2 sleeps 15 seconds Process 1 uses MoveFile() to rename "foo.txt" to "foo2.txt" Process 1 uses MoveFile() to rename "bar.txt" to "foo.txt" Process 1 uses DeleteFile() to remove "foo2.txt" Process 2 awakens and displays "This is FOO!" On the filesystem, we then have: foo.txt containing "This is BAR!" The good news is that this works fine under NT 4 using just MoveFile(). The bad news is that it requires the files be opened using CreateFile() with the FILE_SHARE_DELETE flag set. The C library which ships with Visual C++ 6 ultimately calls CreateFile() via fopen() but with no opportunity through the standard C library routines to use the FILE_SHARE_DELETE flag. And the FILE_SHARE_DELETE flag cannot be used under Windows 95/98 (Bad Parameter). Which means, on those platforms, there still doesn't appear to be a solution. Under NT/XP/2K, AllocateFile() will have to modified to call CreateFile() instead of fopen(). I'm not sure about ME, but I suspect it behaves similarly to 95/98. Mike Mascari mascarm@mascari.com
Stephan Szabo <sszabo@megazone23.bigpanda.com> writes:
> ... let you do the replace and keep reading (at the penalty that
> you've now got to have a way to know when to remove the
> various <something>s)
That is the hard part. Mike's description omitted one crucial step:
6. The old "foo" goes away when the last open file handle for it is
closed.
I doubt there is any practical way for Postgres to cause that to happen
if the OS itself does not have any support for it.
regards, tom lane
I don't think we are not going to be supporting Win9X so there isn't an issue there. We will be supporting Win2000/NT/XP. I don't understand FILE_SHARE_DELETE. I read the description at: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/createfile.asp but I don't understand it: FILE_SHARE_DELETE - Windows NT/2000/XP: Subsequent open operations onthe object will succeed only if delete access is requested. --------------------------------------------------------------------------- Mike Mascari wrote: > Stephan Szabo wrote: > > On Fri, 20 Sep 2002, Mike Mascari wrote: > >> > >>Yes! Indeed that does work. > > > > > > Thinking back, I think that may still fail on Win95 (using MoveFile). > > Once in the past I had to work on (un)installers for Win* and I > > vaguely remember Win95 being more strict than Win98 but that may just > > have been with moving the executable you're currently running. > > Well, here's the test: > > foo.txt contains "This is FOO!" > bar.txt contains "This is BAR!" > > Process 1 opens foo.txt > Process 2 opens foo.txt > Process 1 sleeps 7.5 seconds > Process 2 sleeps 15 seconds > Process 1 uses MoveFile() to rename "foo.txt" to "foo2.txt" > Process 1 uses MoveFile() to rename "bar.txt" to "foo.txt" > Process 1 uses DeleteFile() to remove "foo2.txt" > Process 2 awakens and displays "This is FOO!" > > On the filesystem, we then have: > > foo.txt containing "This is BAR!" > > The good news is that this works fine under NT 4 using just > MoveFile(). The bad news is that it requires the files be opened > using CreateFile() with the FILE_SHARE_DELETE flag set. The C > library which ships with Visual C++ 6 ultimately calls > CreateFile() via fopen() but with no opportunity through the > standard C library routines to use the FILE_SHARE_DELETE flag. > And the FILE_SHARE_DELETE flag cannot be used under Windows > 95/98 (Bad Parameter). Which means, on those platforms, there > still doesn't appear to be a solution. Under NT/XP/2K, > AllocateFile() will have to modified to call CreateFile() > instead of fopen(). I'm not sure about ME, but I suspect it > behaves similarly to 95/98. > > Mike Mascari > mascarm@mascari.com > > > > > > ---------------------------(end of broadcast)--------------------------- > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian wrote: > I don't think we are not going to be supporting Win9X so there isn't an > issue there. We will be supporting Win2000/NT/XP. > > I don't understand FILE_SHARE_DELETE. I read the description at: > > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/createfile.asp > > but I don't understand it: > > FILE_SHARE_DELETE - Windows NT/2000/XP: Subsequent open operations on > the object will succeed only if delete access is requested. I think that's a rather poor description. I think it just means that if the file is opened once via CreateFile() with FILE_SHARE_DELETE, then any subsequent CreateFile() calls will fail unless they too have FILE_SHARE_DELETE. In other words, if one of us can delete this file while its open, any of us can. Mike Mascari mascarm@mascari.com
On Fri, 20 Sep 2002, Mike Mascari wrote: > Bruce Momjian wrote: > > I don't think we are not going to be supporting Win9X so there isn't an > > issue there. We will be supporting Win2000/NT/XP. > > > > I don't understand FILE_SHARE_DELETE. I read the description at: > > > > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/createfile.asp > > > > but I don't understand it: > > > > FILE_SHARE_DELETE - Windows NT/2000/XP: Subsequent open operations on > > the object will succeed only if delete access is requested. > > I think that's a rather poor description. I think it just means > that if the file is opened once via CreateFile() with > FILE_SHARE_DELETE, then any subsequent CreateFile() calls will > fail unless they too have FILE_SHARE_DELETE. In other words, if > one of us can delete this file while its open, any of us can. The question is, what happens if two people have the file open and one goes and tries to delete it? Can the other still read from it?
Mike Mascari wrote: > Bruce Momjian wrote: > > I don't think we are not going to be supporting Win9X so there isn't an > > issue there. We will be supporting Win2000/NT/XP. > > > > I don't understand FILE_SHARE_DELETE. I read the description at: > > > > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/createfile.asp > > > > but I don't understand it: > > > > FILE_SHARE_DELETE - Windows NT/2000/XP: Subsequent open operations on > > the object will succeed only if delete access is requested. > > I think that's a rather poor description. I think it just means > that if the file is opened once via CreateFile() with > FILE_SHARE_DELETE, then any subsequent CreateFile() calls will > fail unless they too have FILE_SHARE_DELETE. In other words, if > one of us can delete this file while its open, any of us can. I don't understand what that gets us. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
It is good that moving the file out of the way works, but it doesn't
completely solve the problem.
What we have now with Unix rename is ideal:
1) old opens continue seeing the old contents2) new opens see the new contents3) the file always exists under the fixed
name
We have that with MoveFileEx(), but we have to loop over the routine
until is succeeds. If we move the old file out of the way, we loose the
ability to know the file always exists and then we have to loop over
open() until is succeeds.
I think we may be best just looping on MoveFileEx() until is succeeds.
We do the pg_pwd writes while holding an exclusive lock on pg_shadow so
that will guarantee that no one else will slip an old version of the
file in after we have written it. However, it also prevents pg_shadow
access while we are doing the looping. Yuck.
---------------------------------------------------------------------------
Mike Mascari wrote:
> Stephan Szabo wrote:
> > On Fri, 20 Sep 2002, Mike Mascari wrote:
> >>Bruce Momjian wrote:
> >>>Mike Mascari wrote:
> >>>>Actually, looking at the pg_pwd code, you want to determine a
> >>>>way for:
> >>>>
> >>>>1. Process 1 opens "foo"
> >>>>2. Process 2 opens "foo"
> >>>>3. Process 1 creates "bar"
> >>>>4. Process 1 renames "bar" to "foo"
> >>>>5. Process 2 can continue to read data from the open file handle
> >>>>and get the original "foo" data.
> >>>
> >>>
> >>>Yep, that's it.
> >>>
> >>
> >>So far, MoveFileEx("foo", "bar", MOVEFILE_REPLACE_EXISTING)
> >>returns "Access Denied" when Process 1 attempts the rename. But
> >>I'm continuing to investigate the possibilities...
> >
> >
> > Does a sequence like
> > Process 1 opens "foo"
> > Process 2 opens "foo"
> > Process 1 creates "bar"
> > Process 1 renames "foo" to <something>
> > - where something is generated to not overlap an existing file
> > Process 1 renames "bar" to "foo"
> > Process 2 continues reading
> > let you do the replace and keep reading (at the penalty that
> > you've now got to have a way to know when to remove the
> > various <something>s)
>
> Yes! Indeed that does work.
>
> Mike Mascari
> mascarm@mascari.com
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>
-- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610)
359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square,
Pennsylvania19073
Mike Mascari wrote: > instead of fopen(). I'm not sure about ME, but I suspect it > behaves similarly to 95/98. I just checked with Katie and the good news (tm) is that the Win32 port we did here at PeerDirect doesn't support 95/98 and ME anyway. It does support NT4, 2000 and XP. So don't bother. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I think we may be best just looping on MoveFileEx() until is succeeds.
> We do the pg_pwd writes while holding an exclusive lock on pg_shadow so
> that will guarantee that no one else will slip an old version of the
> file in after we have written it. However, it also prevents pg_shadow
> access while we are doing the looping. Yuck.
Surely you're not evaluating this on the assumption that the pg_shadow
triggers are the only places that use rename() ?
I see other places in pgstat and relcache that expect rename() to work
per Unix spec.
regards, tom lane
Stephan Szabo wrote:
> On Fri, 20 Sep 2002, Mike Mascari wrote:
>
>
>>I think that's a rather poor description. I think it just means
>>that if the file is opened once via CreateFile() with
>>FILE_SHARE_DELETE, then any subsequent CreateFile() calls will
>>fail unless they too have FILE_SHARE_DELETE. In other words, if
>>one of us can delete this file while its open, any of us can.
>
>
> The question is, what happens if two people have the file open
> and one goes and tries to delete it? Can the other still read
> from it?
Yes. I just tested it and it worked. I'll test Bruce's scenario
as well:
foo contains: "FOO"
bar contains: "BAR"
1. Process 1 opens "foo"
2. Process 2 opens "foo"
3. Process 1 calls MoveFile("foo", "foo2");
4. Process 3 opens "foo" <- Successful?
5. Process 1 calls MoveFile("bar", "foo");
6. Process 4 opens "foo" <- Successful?
7. Process 1 calls DeleteFile("foo2");
8. Process 1, 2, 3, 4 all read from their respective handles.
I think the thing to worry about is a race condition between the
two MoveFile() attempts. A very ugly hack would be to loop in a
CreateFile() in an attempt to open "foo", giving up if the error
is not a NOT EXISTS error code.
Mike Mascari
mascarm@mascari.com
Tom Lane wrote: > Bruce Momjian <pgman@candle.pha.pa.us> writes: > > I think we may be best just looping on MoveFileEx() until is succeeds. > > We do the pg_pwd writes while holding an exclusive lock on pg_shadow so > > that will guarantee that no one else will slip an old version of the > > file in after we have written it. However, it also prevents pg_shadow > > access while we are doing the looping. Yuck. > > Surely you're not evaluating this on the assumption that the pg_shadow > triggers are the only places that use rename() ? > > I see other places in pgstat and relcache that expect rename() to work > per Unix spec. Yes, I know there are others but I think we will need _a_ rename that works 100% and then replace that in all Win32 rename cases. Given what I have seen, I think a single rename with a loop that uses MoveFileEx() may be our best bet. It is localized, doesn't affect the open() code, and should work well. The only downside is that under heavy read activity the loop will loop around a few times but I just don't see another solution. I was initially concerned that the loop in rename could let old renames update the file overwriting newer contents but I realize now that rename() itself has the same issue (an old rename could hit in the code after a newer rename) so in all cases we must already have the proper locking in place. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
I wrote:
> Stephan Szabo wrote:>>
>> The question is, what happens if two people have the file open
>> and one goes and tries to delete it? Can the other still read
>> from it?
>
> Yes. I just tested it and it worked. I'll test Bruce's scenario as well:
>
> foo contains: "FOO"
> bar contains: "BAR"
>
> 1. Process 1 opens "foo"
> 2. Process 2 opens "foo"
> 3. Process 1 calls MoveFile("foo", "foo2");
> 4. Process 3 opens "foo" <- Successful?
> 5. Process 1 calls MoveFile("bar", "foo");
> 6. Process 4 opens "foo" <- Successful?
> 7. Process 1 calls DeleteFile("foo2");
> 8. Process 1, 2, 3, 4 all read from their respective handles.
Process 1: "FOO"
Process 2: "FOO"
Process 3: Error - File does not exist
Process 4: "BAR"
Its interesting in that it allows for Unix-style rename() and
unlink() behavior, but with a race condition. Without Stephan's
two MoveFile() trick and the FILE_SHARE_DELETE flag, however,
the result would be Access Denied. Are the places in the backend
that use rename() and unlink() renaming and unlinking files that
are only opened for a brief moment by other backends?
Mike Mascari
mascarm@mascari.com
Mike Mascari wrote: > Its interesting in that it allows for Unix-style rename() and > unlink() behavior, but with a race condition. Without Stephan's > two MoveFile() trick and the FILE_SHARE_DELETE flag, however, > the result would be Access Denied. Are the places in the backend > that use rename() and unlink() renaming and unlinking files that > are only opened for a brief moment by other backends? Yes, those files are only opened for a brief moment. They are not held open. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Mike Mascari wrote:
> > foo contains: "FOO"
> > bar contains: "BAR"
> >
> > 1. Process 1 opens "foo"
> > 2. Process 2 opens "foo"
> > 3. Process 1 calls MoveFile("foo", "foo2");
> > 4. Process 3 opens "foo" <- Successful?
> > 5. Process 1 calls MoveFile("bar", "foo");
> > 6. Process 4 opens "foo" <- Successful?
> > 7. Process 1 calls DeleteFile("foo2");
> > 8. Process 1, 2, 3, 4 all read from their respective handles.
>
> Process 1: "FOO"
> Process 2: "FOO"
> Process 3: Error - File does not exist
> Process 4: "BAR"
>
> Its interesting in that it allows for Unix-style rename() and
> unlink() behavior, but with a race condition. Without Stephan's
> two MoveFile() trick and the FILE_SHARE_DELETE flag, however,
> the result would be Access Denied. Are the places in the backend
> that use rename() and unlink() renaming and unlinking files that
> are only opened for a brief moment by other backends?
I think we are better off looping over
MoveFileEx(MOVEFILE_REPLACE_EXISTING) until the file isn't opened by
anyone. That localizes the changes to rename only and not out to all
the opens.
The open failure loops when the file isn't there seem much worse.
I am a little concerned about starving the rename when there is a lot of
activity but I don't see a better solution.
-- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610)
359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square,
Pennsylvania19073