Thread: Re: [GENERAL] Linux Largefile Support In Postgresql RPMS

Re: [GENERAL] Linux Largefile Support In Postgresql RPMS

From
Oliver Elphick
Date:
On Tue, 2002-08-13 at 17:11, Rod Taylor wrote:
> > I wouldn't totally discount using NFS for large databases. Believe it or
> > not, with an Oracle database and a Network Appliance for storage, NFS is
> > exactly what is used. We've found that we get better performance with a
> > (properly tuned) NFS mounted NetApp volume than with attached storage on
> > our HPUX box with several 100+GB databases.
>
> We've also tended to keep logs local on raid 1 and the data on a pair of
> custered netapps for PostgreSQL.

But large file support is not really an issue for the database itself,
since table files are split at 1Gb.  Unless that changes, the database
is not a problem.

--
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight, UK
http://www.lfix.co.uk/oliver
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
                 ========================================
     "Watch ye therefore, and pray always, that ye may be
      accounted worthy to escape all these things that shall
      come to pass, and to stand before the Son of man."
                               Luke 21:36


Re: [GENERAL] Linux Largefile Support In Postgresql RPMS

From
Tom Lane
Date:
Oliver Elphick <olly@lfix.co.uk> writes:
> But large file support is not really an issue for the database itself,
> since table files are split at 1Gb.  Unless that changes, the database
> is not a problem.

I see no really good reason to change the file-split logic.  The places
where the backend might possibly need large-file support are
    * backend-side COPY to or from a large file
    * postmaster log to stderr --- does this fail if log output
      exceeds 2G?
There might be some other similar issues, but that's all that comes to
mind offhand.

On a system where building with large-file support is reasonably
standard, I agree that PG should be built that way too.  Where it's
not so standard, I agree with Andrew Sullivan's concerns ...

            regards, tom lane

Re: [GENERAL] Linux Largefile Support In Postgresql RPMS

From
Andrew Sullivan
Date:
On Tue, Aug 13, 2002 at 01:04:02PM -0400, Tom Lane wrote:
>
> I see no really good reason to change the file-split logic.  The places
> where the backend might possibly need large-file support are
>     * backend-side COPY to or from a large file

I _think_ this causes a crash.  At least, I _think_ that's what
caused it one day (I was doing one of those jackhammer-the-server
sorts of tests, and it was one of about 50 things I was doing at the
time, to see if I could make it fall over.  I did, but not where I
expected, and way beyond any real load we could anticipate).

>     * postmaster log to stderr --- does this fail if log output
>       exceeds 2G?

Yes, definitely, at least on Solaris.

A

--
----
Andrew Sullivan                               87 Mowat Avenue
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M6K 3E3
                                         +1 416 646 3304 x110


Re: [GENERAL] Linux Largefile Support In Postgresql RPMS

From
strange@nsk.yi.org
Date:
On Tue, Aug 13, 2002 at 01:04:02PM -0400, Tom Lane wrote:
> On a system where building with large-file support is reasonably
> standard, I agree that PG should be built that way too.  Where it's
> not so standard, I agree with Andrew Sullivan's concerns ...

What do you mean by "standard"? That only some filesystems are supported?
In Linux the vfat filesystem doesn't support largefiles, so the behaviour
is the same as if the application didn't specify O_LARGEFILE to open(2):
As Helge Bahmann pointed out, "kernel will refuse to write files larger than
2GB". In current Linux, a signal (SIGXFSZ) is sent to the application
that then dumps core.


So, the use of O_LARGEFILE is nullified by the lack of support by the
filesystem, but no problem is introduced by the application supporting
largefiles, it already existed before.

All the crashes and problems presented on these lists occur when largefile
support isn't compiled, I didn't see one occuring from any application
having the support, but not the filesystem. (Your "not so standard
support"?)

The changes to postgresql doesn't seem complicated, I can try to make them
myself (fcntl on stdout, stdin; add check to autoconf; etc.) if no one
else volunteers.

Regards,
Luciano Rocha

--
Consciousness: that annoying time between naps.

Re: [GENERAL] Linux Largefile Support In Postgresql RPMS

From
Andrew Sullivan
Date:
On Tue, Aug 13, 2002 at 06:45:59PM +0100, strange@nsk.yi.org wrote:

> support isn't compiled, I didn't see one occuring from any application
> having the support, but not the filesystem. (Your "not so standard

Wrong.  The symptom is _exactly the same_ if the program doesn't have
the support, the filesystem doesn't have the support, or both, at
least on Solaris.  I've checked.

A

--
----
Andrew Sullivan                               87 Mowat Avenue
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M6K 3E3
                                         +1 416 646 3304 x110


Re: [GENERAL] Linux Largefile Support In Postgresql RPMS

From
Greg Copeland
Date:
On Tue, 2002-08-13 at 12:45, strange@nsk.yi.org wrote:
> On Tue, Aug 13, 2002 at 01:04:02PM -0400, Tom Lane wrote:
> > On a system where building with large-file support is reasonably
> > standard, I agree that PG should be built that way too.  Where it's
> > not so standard, I agree with Andrew Sullivan's concerns ...
>
> What do you mean by "standard"? That only some filesystems are supported?
> In Linux the vfat filesystem doesn't support largefiles, so the behaviour
> is the same as if the application didn't specify O_LARGEFILE to open(2):
> As Helge Bahmann pointed out, "kernel will refuse to write files larger than
> 2GB". In current Linux, a signal (SIGXFSZ) is sent to the application
> that then dumps core.
>
>
> So, the use of O_LARGEFILE is nullified by the lack of support by the
> filesystem, but no problem is introduced by the application supporting
> largefiles, it already existed before.
>

Thank you.  That's a point that I previously pointed out...you just did
a much better job of it.  Specifically, want to stress that enabling
large file support is not dangerous.

Greg


Attachment

Re: [GENERAL] Linux Largefile Support In Postgresql RPMS

From
Greg Copeland
Date:
On Tue, 2002-08-13 at 12:04, Tom Lane wrote:

> On a system where building with large-file support is reasonably
> standard, I agree that PG should be built that way too.  Where it's
> not so standard, I agree with Andrew Sullivan's concerns ...


Agreed.  This is what I originally asked for.

Greg


Attachment

Re: [GENERAL] Linux Largefile Support In Postgresql RPMS

From
strange@nsk.yi.org
Date:
On Tue, Aug 13, 2002 at 02:09:07PM -0400, Andrew Sullivan wrote:
> On Tue, Aug 13, 2002 at 06:45:59PM +0100, strange@nsk.yi.org wrote:
>
> > support isn't compiled, I didn't see one occuring from any application
> > having the support, but not the filesystem. (Your "not so standard
>
> Wrong.  The symptom is _exactly the same_ if the program doesn't have
> the support, the filesystem doesn't have the support, or both, at
> least on Solaris.  I've checked.

??

My point is that: Having postgresql the support doesn't bring NEW errors.

I never said postgresql would automagically gain support on filesystems
that don't support largfiles, I said no one mentioned an error caused by
postgresql *having* the support, but *not the filesystem*. Maybe I wasn't
clear, but I meant *new* errors.

As it seams, adding support to largefiles doesn't break anything.

Regards,
Luciano Rocha

--
Consciousness: that annoying time between naps.

Re: [GENERAL] Linux Largefile Support In Postgresql RPMS

From
Peter Eisentraut
Date:
Tom Lane writes:

>     * postmaster log to stderr --- does this fail if log output
>       exceeds 2G?

That would be an issue of the shell, not the postmaster.

-- 
Peter Eisentraut   peter_e@gmx.net