Thread: Re: [GENERAL] Linux Largefile Support In Postgresql RPMS
On Tue, 2002-08-13 at 17:11, Rod Taylor wrote: > > I wouldn't totally discount using NFS for large databases. Believe it or > > not, with an Oracle database and a Network Appliance for storage, NFS is > > exactly what is used. We've found that we get better performance with a > > (properly tuned) NFS mounted NetApp volume than with attached storage on > > our HPUX box with several 100+GB databases. > > We've also tended to keep logs local on raid 1 and the data on a pair of > custered netapps for PostgreSQL. But large file support is not really an issue for the database itself, since table files are split at 1Gb. Unless that changes, the database is not a problem. -- Oliver Elphick Oliver.Elphick@lfix.co.uk Isle of Wight, UK http://www.lfix.co.uk/oliver GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839 932A 614D 4C34 3E1D 0C1C ======================================== "Watch ye therefore, and pray always, that ye may be accounted worthy to escape all these things that shall come to pass, and to stand before the Son of man." Luke 21:36
Oliver Elphick <olly@lfix.co.uk> writes:
> But large file support is not really an issue for the database itself,
> since table files are split at 1Gb. Unless that changes, the database
> is not a problem.
I see no really good reason to change the file-split logic. The places
where the backend might possibly need large-file support are
* backend-side COPY to or from a large file
* postmaster log to stderr --- does this fail if log output
exceeds 2G?
There might be some other similar issues, but that's all that comes to
mind offhand.
On a system where building with large-file support is reasonably
standard, I agree that PG should be built that way too. Where it's
not so standard, I agree with Andrew Sullivan's concerns ...
regards, tom lane
On Tue, Aug 13, 2002 at 01:04:02PM -0400, Tom Lane wrote:
>
> I see no really good reason to change the file-split logic. The places
> where the backend might possibly need large-file support are
> * backend-side COPY to or from a large file
I _think_ this causes a crash. At least, I _think_ that's what
caused it one day (I was doing one of those jackhammer-the-server
sorts of tests, and it was one of about 50 things I was doing at the
time, to see if I could make it fall over. I did, but not where I
expected, and way beyond any real load we could anticipate).
> * postmaster log to stderr --- does this fail if log output
> exceeds 2G?
Yes, definitely, at least on Solaris.
A
--
----
Andrew Sullivan 87 Mowat Avenue
Liberty RMS Toronto, Ontario Canada
<andrew@libertyrms.info> M6K 3E3
+1 416 646 3304 x110
On Tue, Aug 13, 2002 at 01:04:02PM -0400, Tom Lane wrote: > On a system where building with large-file support is reasonably > standard, I agree that PG should be built that way too. Where it's > not so standard, I agree with Andrew Sullivan's concerns ... What do you mean by "standard"? That only some filesystems are supported? In Linux the vfat filesystem doesn't support largefiles, so the behaviour is the same as if the application didn't specify O_LARGEFILE to open(2): As Helge Bahmann pointed out, "kernel will refuse to write files larger than 2GB". In current Linux, a signal (SIGXFSZ) is sent to the application that then dumps core. So, the use of O_LARGEFILE is nullified by the lack of support by the filesystem, but no problem is introduced by the application supporting largefiles, it already existed before. All the crashes and problems presented on these lists occur when largefile support isn't compiled, I didn't see one occuring from any application having the support, but not the filesystem. (Your "not so standard support"?) The changes to postgresql doesn't seem complicated, I can try to make them myself (fcntl on stdout, stdin; add check to autoconf; etc.) if no one else volunteers. Regards, Luciano Rocha -- Consciousness: that annoying time between naps.
On Tue, Aug 13, 2002 at 06:45:59PM +0100, strange@nsk.yi.org wrote:
> support isn't compiled, I didn't see one occuring from any application
> having the support, but not the filesystem. (Your "not so standard
Wrong. The symptom is _exactly the same_ if the program doesn't have
the support, the filesystem doesn't have the support, or both, at
least on Solaris. I've checked.
A
--
----
Andrew Sullivan 87 Mowat Avenue
Liberty RMS Toronto, Ontario Canada
<andrew@libertyrms.info> M6K 3E3
+1 416 646 3304 x110
On Tue, 2002-08-13 at 12:45, strange@nsk.yi.org wrote: > On Tue, Aug 13, 2002 at 01:04:02PM -0400, Tom Lane wrote: > > On a system where building with large-file support is reasonably > > standard, I agree that PG should be built that way too. Where it's > > not so standard, I agree with Andrew Sullivan's concerns ... > > What do you mean by "standard"? That only some filesystems are supported? > In Linux the vfat filesystem doesn't support largefiles, so the behaviour > is the same as if the application didn't specify O_LARGEFILE to open(2): > As Helge Bahmann pointed out, "kernel will refuse to write files larger than > 2GB". In current Linux, a signal (SIGXFSZ) is sent to the application > that then dumps core. > > > So, the use of O_LARGEFILE is nullified by the lack of support by the > filesystem, but no problem is introduced by the application supporting > largefiles, it already existed before. > Thank you. That's a point that I previously pointed out...you just did a much better job of it. Specifically, want to stress that enabling large file support is not dangerous. Greg
Attachment
On Tue, 2002-08-13 at 12:04, Tom Lane wrote: > On a system where building with large-file support is reasonably > standard, I agree that PG should be built that way too. Where it's > not so standard, I agree with Andrew Sullivan's concerns ... Agreed. This is what I originally asked for. Greg
Attachment
On Tue, Aug 13, 2002 at 02:09:07PM -0400, Andrew Sullivan wrote: > On Tue, Aug 13, 2002 at 06:45:59PM +0100, strange@nsk.yi.org wrote: > > > support isn't compiled, I didn't see one occuring from any application > > having the support, but not the filesystem. (Your "not so standard > > Wrong. The symptom is _exactly the same_ if the program doesn't have > the support, the filesystem doesn't have the support, or both, at > least on Solaris. I've checked. ?? My point is that: Having postgresql the support doesn't bring NEW errors. I never said postgresql would automagically gain support on filesystems that don't support largfiles, I said no one mentioned an error caused by postgresql *having* the support, but *not the filesystem*. Maybe I wasn't clear, but I meant *new* errors. As it seams, adding support to largefiles doesn't break anything. Regards, Luciano Rocha -- Consciousness: that annoying time between naps.
Tom Lane writes: > * postmaster log to stderr --- does this fail if log output > exceeds 2G? That would be an issue of the shell, not the postmaster. -- Peter Eisentraut peter_e@gmx.net