Thread: True ACID under linux (no fsync)?
Hi, Just a quickie - I heard that linux does not have a working fsync() call (it has no concept of raw devices). In other words, fsync cannot be used to ensure that data is on disk (try it with a floppy - it returns IMMEDIATELY!!! - long before the write finishes) This is for ALL linuxes. Does this mean that postgres is unrealiable (or non ACID-conforming) under linux? Gary -- Friends help you move. Real friends help you move bodies. _______________________________________________________________________ Free Unlimited Internet Access! Try it now! http://www.zdnet.com/downloads/altavista/index.html _______________________________________________________________________
Strange. I did a test on Redhat Linux 7.0 with fsync() turned off and my batchfile of inserts ran 10 times as fast. But that was on an slow PC with a slow IDE harddrive. What Linux distribution and kernel version are you running? Poul L. Christiansen "Gary Howland (During daytime)" wrote: > > Hi, > > Just a quickie - I heard that linux does not have a working fsync() call > (it has no concept of raw devices). In other words, fsync cannot be used > to ensure that data is on disk (try it with a floppy - it returns IMMEDIATELY!!! - long before the write finishes) > > This is for ALL linuxes. > > Does this mean that postgres is unrealiable (or non ACID-conforming) under linux? > > Gary > > -- > Friends help you move. > Real friends help you move bodies. > > _______________________________________________________________________ > > Free Unlimited Internet Access! Try it now! > http://www.zdnet.com/downloads/altavista/index.html > > _______________________________________________________________________
I must have missed that one. How do you turn off fsync()?? I have lots of batch loading to do during which ACID is of no use. ++ kevin "Poul L. Christiansen" wrote: > > Strange. > > I did a test on Redhat Linux 7.0 with fsync() turned off and my > batchfile of inserts ran 10 times as fast. But that was on an slow PC > with a slow IDE harddrive. > > What Linux distribution and kernel version are you running? > > Poul L. Christiansen > > "Gary Howland (During daytime)" wrote: > > > > Hi, > > > > Just a quickie - I heard that linux does not have a working fsync() call > > (it has no concept of raw devices). In other words, fsync cannot be used > > to ensure that data is on disk (try it with a floppy - it returns IMMEDIATELY!!! - long before the write finishes) > > > > This is for ALL linuxes. > > > > Does this mean that postgres is unrealiable (or non ACID-conforming) under linux? > > > > Gary > > > > -- > > Friends help you move. > > Real friends help you move bodies. > > > > _______________________________________________________________________ > > > > Free Unlimited Internet Access! Try it now! > > http://www.zdnet.com/downloads/altavista/index.html > > > > _______________________________________________________________________ -- Kevin O'Gorman (805) 650-6274 mailto:kogorman@pacbell.net Permanent e-mail forwarder: mailto:Kevin.O'Gorman.64@Alum.Dartmouth.org At school: mailto:kogorman@cs.ucsb.edu Web: http://www.cs.ucsb.edu/~kogorman/index.html Web: http://trixie.kosman.via.ayuda.com/~kevin/index.html "There is a freedom lying beyond circumstance, derived from the direct intuition that life can be grounded upon its absorption in what is changeless amid change" -- Alfred North Whitehead
Use the "-F" option. I start PostgreSQL with this line: su -l postgres -c "/usr/bin/pg_ctl -D $PGDATA -p /usr/bin/postmaster -o '-i -B 4096 -o -F' start >/dev/null 2>&1" < /dev/null Poul L. Christiansen Kevin O'Gorman wrote: > > I must have missed that one. How do you turn off fsync()?? > I have lots of batch loading to do during which ACID is of no use. > > ++ kevin > > "Poul L. Christiansen" wrote: > > > > Strange. > > > > I did a test on Redhat Linux 7.0 with fsync() turned off and my > > batchfile of inserts ran 10 times as fast. But that was on an slow PC > > with a slow IDE harddrive. > > > > What Linux distribution and kernel version are you running? > > > > Poul L. Christiansen > > > > "Gary Howland (During daytime)" wrote: > > > > > > Hi, > > > > > > Just a quickie - I heard that linux does not have a working fsync() call > > > (it has no concept of raw devices). In other words, fsync cannot be used > > > to ensure that data is on disk (try it with a floppy - it returns IMMEDIATELY!!! - long before the write finishes) > > > > > > This is for ALL linuxes. > > > > > > Does this mean that postgres is unrealiable (or non ACID-conforming) under linux? > > > > > > Gary > > > > > > -- > > > Friends help you move. > > > Real friends help you move bodies. > > > > > > _______________________________________________________________________ > > > > > > Free Unlimited Internet Access! Try it now! > > > http://www.zdnet.com/downloads/altavista/index.html > > > > > > _______________________________________________________________________ > > -- > Kevin O'Gorman (805) 650-6274 mailto:kogorman@pacbell.net > Permanent e-mail forwarder: mailto:Kevin.O'Gorman.64@Alum.Dartmouth.org > At school: mailto:kogorman@cs.ucsb.edu > Web: http://www.cs.ucsb.edu/~kogorman/index.html > Web: http://trixie.kosman.via.ayuda.com/~kevin/index.html > > "There is a freedom lying beyond circumstance, > derived from the direct intuition that life can > be grounded upon its absorption in what is > changeless amid change" > -- Alfred North Whitehead
> Use the "-F" option. > > I start PostgreSQL with this line: > su -l postgres -c "/usr/bin/pg_ctl -D $PGDATA -p /usr/bin/postmaster -o > '-i -B 4096 -o -F' start >/dev/null 2>&1" < /dev/null Interesting... trying to start postgres 7.0.2 with the "-F" simply returns a list of options to me, as if it can't understand it. It sure does sound appealing to me, though. With a few hundred megs of disk cache (between the kernel and the RAID controller), I've always wished that PostgreSQL would take advantage of it when writing. : ) steve
On Tue, 31 Oct 2000, Steve Wolfe wrote:
> Interesting... trying to start postgres 7.0.2 with the "-F" simply returns
> a list of options to me, as if it can't understand it. It sure does sound
> appealing to me, though. With a few hundred megs of disk cache (between the
> kernel and the RAID controller), I've always wished that PostgreSQL would
> take advantage of it when writing. : )
postmaster -o -F -D /your/datadirectory
The -o tells postmaster to pass -F as backend option.
man postmaster and man postgres for more information.
--
Dominic J. Eidson
"Baruk Khazad! Khazad ai-menu!" - Gimli
-------------------------------------------------------------------------------
http://www.the-infinite.org/ http://www.the-infinite.org/~dominic/
"Gary Howland (During daytime)" wrote: > Hi, > > Just a quickie - I heard that linux does not have a working fsync() call > (it has no concept of raw devices). In other words, fsync cannot be used > to ensure that data is on disk (try it with a floppy - it returns IMMEDIATELY!!! - long before the write finishes) Are you sure? Where is this information? > This is for ALL linuxes. > > Does this mean that postgres is unrealiable (or non ACID-conforming) under linux? > > Gary > > -- > Friends help you move. > Real friends help you move bodies. > > _______________________________________________________________________ > > Free Unlimited Internet Access! Try it now! > http://www.zdnet.com/downloads/altavista/index.html > > _______________________________________________________________________
I may be horriblly out of my element here but, I'm going to jump in anyway :) If you mount a floppy and copy a large file to it, you will get a prompt back fairly quickly. If you type: sync right after The sync command writes everything to the floppy as expected. Shawn On Tue, 31 Oct 2000, markw wrote: > "Gary Howland (During daytime)" wrote: > > > Hi, > > > > Just a quickie - I heard that linux does not have a working fsync() call > > (it has no concept of raw devices). In other words, fsync cannot be used > > to ensure that data is on disk (try it with a floppy - it returns IMMEDIATELY!!! - long before the write finishes) > > Are you sure? Where is this information? > > > > This is for ALL linuxes. > > > > Does this mean that postgres is unrealiable (or non ACID-conforming) under linux? > > > > Gary > > > > -- > > Friends help you move. > > Real friends help you move bodies. > > > > _______________________________________________________________________ > > > > Free Unlimited Internet Access! Try it now! > > http://www.zdnet.com/downloads/altavista/index.html > > > > _______________________________________________________________________ > > >
"Gary Howland (During daytime)" wrote:
> Just a quickie - I heard that linux does not have a working fsync()
> call (it has no concept of raw devices). In other words, fsync cannot
> be used to ensure that data is on disk (try it with a floppy - it
> returns IMMEDIATELY!!! - long before the write finishes)
That's because the mounted drive is buffered in memory, so when you copy a
file, you are copying it to memory (which is very fast), and the kernel
implicitly syncs with the media when the buffer fills or when an explicit
sync is done or when it is unmounted.
If there is no working fsync (and fdatasync), why do the Linux man pages
claim it to be POSIX.1b compliant?
Can you back up your claim with actual documentation? Saying 'I heard'
puts it in the realm of urban legendry...
Brett W. McCoy
http://www.chapelperilous.net
---------------------------------------------------------------------------
This is the ____LAST time I take travel suggestions from Ray Bradbury!
shawn everett wrote: > I may be horriblly out of my element here but, I'm going to jump in anyway > :) > > If you mount a floppy and copy a large file to it, you will get a prompt > back fairly quickly. > > If you type: sync right after > > The sync command writes everything to the floppy as expected. Yes, but does cp call fsync()? That is the question. I would suspect that it does not. A non-functional fsync call is a serious error, and one should think more people would mention it. This is the first I have heard. > > > Shawn > > On Tue, 31 Oct 2000, markw wrote: > > > "Gary Howland (During daytime)" wrote: > > > > > Hi, > > > > > > Just a quickie - I heard that linux does not have a working fsync() call > > > (it has no concept of raw devices). In other words, fsync cannot be used > > > to ensure that data is on disk (try it with a floppy - it returns IMMEDIATELY!!! - long before the write finishes) > > > > Are you sure? Where is this information? > > > > > > > This is for ALL linuxes. > > > > > > Does this mean that postgres is unrealiable (or non ACID-conforming) under linux? > > > > > > Gary > > > > > > -- > > > Friends help you move. > > > Real friends help you move bodies. > > > > > > _______________________________________________________________________ > > > > > > Free Unlimited Internet Access! Try it now! > > > http://www.zdnet.com/downloads/altavista/index.html > > > > > > _______________________________________________________________________ > > > > > >
* bmccoy@chapelperilous.net <bmccoy@chapelperilous.net> [001031 12:28] wrote: > "Gary Howland (During daytime)" wrote: > > > Just a quickie - I heard that linux does not have a working fsync() > > call (it has no concept of raw devices). In other words, fsync cannot > > be used to ensure that data is on disk (try it with a floppy - it > > returns IMMEDIATELY!!! - long before the write finishes) > > That's because the mounted drive is buffered in memory, so when you copy a > file, you are copying it to memory (which is very fast), and the kernel > implicitly syncs with the media when the buffer fills or when an explicit > sync is done or when it is unmounted. > > If there is no working fsync (and fdatasync), why do the Linux man pages > claim it to be POSIX.1b compliant? > > Can you back up your claim with actual documentation? Saying 'I heard' > puts it in the realm of urban legendry... You guys are missing the difference between fsync(2) and sync(1), on Linux fsync(2) (called on a file descriptor) WILL sync out all the data halfway to specification, HOWEVER afaik it will not fsync out all directory entries that may contain this file. If this is still true, that means that on Linux you must walk '..' calling fsync on each directory above the file before considering the file really there. Now sync(2) and sync(1) (the userland interface to the sync(2) syscall) work beyond the spec, on Linux sync(2) shouldn't return until all data has been sync'd to disk, however the standard only says that sync(2) should schedule the data to be sync'd out, not that it needs to wait until it's all done. <blatant plug> Of course FreeBSD fsync(2) does properly walk '..' for you and sync(2) behaves as specified by spec, mearly scheduling the IO, not doing it syncronously. </blatant plug> -- -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] "I have the heart of a child; I keep it in a jar on my desk."
On Tue, 31 Oct 2000, Alfred Perlstein wrote: > You guys are missing the difference between fsync(2) and sync(1), > on Linux fsync(2) (called on a file descriptor) WILL sync out all > the data halfway to specification, HOWEVER afaik it will not fsync > out all directory entries that may contain this file. That all depends on a filesystem. To my belief, all post-ext2 filesystems (such as tux2, reiserfs, xfs) will properly flush metadata when necessary.. -alex
On 31 Oct 2000, Gary Howland wrote:
> Just a quickie - I heard that linux does not have a working fsync() call
At least the manpage for fsync says that it does.
The implementation:
/* .. finally sync the buffers to disk */
dev = inode->i_dev;
return sync_buffers(dev, 1);
It really looks like it IS implemented. But probably on Linux not just the
file data/metadata is synced, also all that device's data, which makes it
very inefficient, but presumably `safe'.
NB: don't forget that fsync() merely ensures that data was sent to the
disk controller. Maybe this one has a cache (e.g. a fast SCSI harddrive),
and if power fails, well. If fsync() was calling the SCSI FLUSH command,
maybe that could be done, but that would not just sync the file.
On Wed, Nov 01, 2000 at 11:16:24AM +0100, Marc SCHAEFER wrote: > > Just a quickie - I heard that linux does not have a working fsync() call > At least the manpage for fsync says that it does. The Linux fsync call (at least on ext2, I don't know if it's a filesystem semantics issue or a VFS semantics issue) works slightly different than the BSD call: On Linux, fsync flushes all the file data and *file* metadata, but doesn't handle the directory metadata. Directories are treated as seperate files which can be fsync'd seperately, and need to be if you're adding or removing files synchronously. -- Bruce Guenter <bruceg@em.ca> http://em.ca/~bruceg/