Thread: Large files on linux
Dear all: I am having trouble with large files on a Linux box (RH 6.2). I know there is a limit of 2 GB on the file size, but do not know if this is kernel related, filesystem related or both. I am researchiing the issue (having some trouble to find info) and found that there are several alternatives. Can you give me advice for/against the ones you think are best/worse. Also, if you happen to have links to other places which might have information on this issue, please pass them along. Here are my options: i) patch the kernel. I didn't find much information about this one. I don't know yet whether this is a kernel related limitation or filesystem-related or both. I have also read that programs had to be patched to use this 'large_file' patch. Will probably avoid this one. ii) try a new kernel, 2.4-test have to do some research into this yet, but apparently the new linux kernel will support large files. My understanding right now is that ext2fs do not support files this large, so this option will probably go together with the next one. iii) try a new filesystem. several filesystems appear to support large files (but not ext2fs, and apparently not ext3fs). (http://www.linuxgazette.com/issue55/florido.html). iv) go for another distro or unix SuSE 7.0 reportedly has support for large files (> 2GB) although it uses a pre2.4 kernel. I don't know how SuSE does it (patches like the ones mentioned in (1) or a different filesystem?). Another option is to go for a different unix, like FreeBSD, which has had support for large files for a long while. Thanks in advance, Fernan
Fernan Aguero <fernan@iib.unsam.edu.ar> writes: > I am having trouble with large files on a Linux box (RH 6.2). I know there > is a limit of 2 GB on the file size, but do not know if this is kernel > related, filesystem related or both. > > I am researchiing the issue (having some trouble to find info) and found > that there are several alternatives. Can you give me advice for/against the > ones you think are best/worse. Also, if you happen to have links to other > places which might have information on this issue, please pass them along. > > Here are my options: > > i) patch the kernel. > I didn't find much information about this one. I don't know yet whether > this is a kernel related limitation or filesystem-related or both. I have > also read that programs had to be patched to use this 'large_file' patch. > Will probably avoid this one. > > ii) try a new kernel, 2.4-test > have to do some research into this yet, but apparently the new linux kernel > will support large files. My understanding right now is that ext2fs do not > support files this large, It does. > iii) try a new filesystem. > several filesystems appear to support large files (but not ext2fs, and > apparently not ext3fs) On 2.2, it's VFS related. ext3/ext2 doesn't have such problems. Red Hat Linux 7.0 includes a kernel (the "enterprise" one) with the LFS patched included and test. It's also ready for 2.4 -- Trond Eivind Glomsrød Red Hat, Inc.
* Fernan Aguero <fernan@iib.unsam.edu.ar> [001211 12:45] wrote: > Dear all: > > I am having trouble with large files on a Linux box (RH 6.2). I know there > is a limit of 2 GB on the file size, but do not know if this is kernel > related, filesystem related or both. Afaik it's both. Honestly Postgresql should be able to deal with this limitation by using more than one file per table. But if you really want to support large files on a free UNIX, I'd try FreeBSD. best of luck, -- -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] "I have the heart of a child; I keep it in a jar on my desk."
Fernan Aguero writes: > I am having trouble with large files on a Linux box (RH 6.2). I know there > is a limit of 2 GB on the file size, ...but that doesn't affect table size, database size, or whatever you're thinking of. -- Peter Eisentraut peter_e@gmx.net http://yi.org/peter-e/
Alfred Perlstein schrieb:
> * Fernan Aguero <fernan@iib.unsam.edu.ar> [001211 12:45] wrote:
> > Dear all:
> >
> > I am having trouble with large files on a Linux box (RH 6.2). I know there
> > is a limit of 2 GB on the file size, but do not know if this is kernel
> > related, filesystem related or both.
>
> Afaik it's both.
>
> Honestly Postgresql should be able to deal with this limitation by
> using more than one file per table.
>
> But if you really want to support large files on a free UNIX,
> I'd try FreeBSD.
>
> best of luck,
> --
> -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]
> "I have the heart of a child; I keep it in a jar on my desk."
Kernel >= 2.4 can manage large files. As far as I have tried this it works
perfectly well.
Hans
Peter Eisentraut wrote: > Fernan Aguero writes: > > I am having trouble with large files on a Linux box (RH 6.2). I know there > > is a limit of 2 GB on the file size, > ...but that doesn't affect table size, database size, or whatever you're > thinking of. Nope, PostgreSQL segments nicely for tables. But, unless you do chunking, it _does_ affect dumpfile size. Someone posted awhile back a script that did dumpchunking. Should be in the archives. -- Lamar Owen WGCR Internet Radio 1 Peter 4:11
Quoting Fernan Aguero <fernan@iib.unsam.edu.ar>: > Dear all: > > I am having trouble with large files on a Linux box (RH 6.2). I know > there > is a limit of 2 GB on the file size, but do not know if this is kernel > related, filesystem related or both. IIRC Partly Kernel, Partly FileSystem. For large tables we split the tables at the 1Gb point. The Storage Manager then deals with which file it needs to open when it knows that the file position is beyond any one 1Gb point. NB: The source was doing this at the 2Gb point but (6.3.x or 6.4.x I think) I did some tests for another project and found that Linux didn't like files exactly 2Gb in size. After discussion here, 1Gb was chosen as a nice round figure. -- Peter Mount peter@retep.org.uk PostgreSQL JDBC Driver: http://www.retep.org.uk/postgres/ RetepPDF PDF library for Java: http://www.retep.org.uk/pdf/
>Bruce Momjian wrote: > >This has nothing to do with PostgreSQL. We don't need large files to go >over 4gig tables. We split them on our own. >Peter Eisentraut wrote: > >But, unless you do chunking, it _does_ affect dumpfile size. Someone >posted awhile back a script that did dumpchunking. Should be in the >archives. >-- >Lamar Owen >WGCR Internet Radio >1 Peter 4:11 Sorry for not being clear in my first post. The problems I am having are when dumping data, as Peter rightly guessed. I remembered reading something about this on the list, that's why i asked my questions here. Although I considered writing a script to divide the dumped data into smaller chunks, i (for other reasons) preferred to go for a cleaner solution, and that is having a system able to cope with large files. Thanks to all for your replies. Here's a brief summary of things: Linux kernel 2.4 supports large files. As mentioned, SuSE and RedHat, both at version 7.0 come with patched 2.2.x kernels that support large files. SuSE uses the ReiserFS, while RedHat uses ext2fs. Since I'm using RH 6.2, the best thing to do is to update to 7.0 and see if that helps. BTW, RH 7.0 is ready for 2.4 kernels (Thanks Trond) -- Lic. Fernan Aguero Tel: (54-11) 4752-0021 Instituto de Investigaciones Biotecnologicas Fax: (54-11) 4752-9639 Universidad Nacional de General San Martin