Thread: Rather incorrect text in admin guide

Rather incorrect text in admin guide

From
Michael Graff
Date:
In the admin guide, under the section "Large Databases" is the
following paragraph:

    Since Postgres allows tables larger than the maximum file size
    on your system, it can be problematic to dump the table to a
    file, since the resulting file will likely be larger than the
    maximum size allowed by your system. As pg_dump writes to the
    standard output, you can just use standard *nix tools to work
    around this possible problem.

This is a generalization of, most likely, a failing in linux.

NetBSD (which I use) will allow file sizes up to 2^64 -- I don't think
anyone has generated a postgresql database that large yet.

You might want to qualify that with "Operating systems which support
64-bit file sizes (such as NetBSD) will have no problem with large
databases" or "some operating systems are limited to 2-gigabyte files
(such as linux)"

--Michael

Re: Rather incorrect text in admin guide

From
Tom Samplonius
Date:
On 27 Dec 2000, Michael Graff wrote:

> In the admin guide, under the section "Large Databases" is the
> following paragraph:
>
>     Since Postgres allows tables larger than the maximum file size
>     on your system, it can be problematic to dump the table to a
>     file, since the resulting file will likely be larger than the
>     maximum size allowed by your system. As pg_dump writes to the
>     standard output, you can just use standard *nix tools to work
>     around this possible problem.
>
> This is a generalization of, most likely, a failing in linux.
>
> NetBSD (which I use) will allow file sizes up to 2^64 -- I don't think
> anyone has generated a postgresql database that large yet.
>
> You might want to qualify that with "Operating systems which support
> 64-bit file sizes (such as NetBSD) will have no problem with large
> databases" or "some operating systems are limited to 2-gigabyte files
> (such as linux)"

  Actually, it much stranger than that.  The ext2fs filesystem can store
large files, but the filesystem layer on i386 will not.  The filesystem
layer on Alphas can.  Apparently, the worry was that 64 point pointers
would slow the i386 version down too much since it is only a 32 bit CPU.
However, the various xBSD flavours do support large files, and so does
Solaris, on all platforms they support.

  Also NFSv2 is limited to 2GB, even if the client and server have no
issues.  It is a protocol thing that is fixed in NFSv3.  I doubt that
anyone is putting their postgres databases on a NFS server, but you never
know.

> --Michael


Tom


Re: Rather incorrect text in admin guide

From
Hannu Krosing
Date:
Michael Graff wrote:
>
> In the admin guide, under the section "Large Databases" is the
> following paragraph:
>
>         Since Postgres allows tables larger than the maximum file size
>         on your system, it can be problematic to dump the table to a
>         file, since the resulting file will likely be larger than the
>         maximum size allowed by your system. As pg_dump writes to the
>         standard output, you can just use standard *nix tools to work
>         around this possible problem.
>
> This is a generalization of, most likely, a failing in linux.
>
> NetBSD (which I use) will allow file sizes up to 2^64 -- I don't think
> anyone has generated a postgresql database that large yet.
>
> You might want to qualify that with "Operating systems which support
> 64-bit file sizes (such as NetBSD) will have no problem with large
> databases" or "some operating systems are limited to 2-gigabyte files
> (such as linux)"

Or more correctly "(such as some versions of linux)"

---------
Hannu

Re: Rather incorrect text in admin guide

From
Peter Eisentraut
Date:
Michael Graff writes:

> In the admin guide, under the section "Large Databases" is the
> following paragraph:
>
>     Since Postgres allows tables larger than the maximum file size
>     on your system, it can be problematic to dump the table to a
>     file, since the resulting file will likely be larger than the
>     maximum size allowed by your system. As pg_dump writes to the
>     standard output, you can just use standard *nix tools to work
>     around this possible problem.
>
> This is a generalization of, most likely, a failing in linux.

The actual maximum file size of a system at hand is irrelevant to the
correctness of the cited paragraph.  Postgres will still allow tables
larger than the maximum file size and the illustrated approaches to
dumping still apply.

> NetBSD (which I use) will allow file sizes up to 2^64 -- I don't think
> anyone has generated a postgresql database that large yet.

Possibly, but how is this relevant to what *could* be done?

--
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/


Re: Rather incorrect text in admin guide

From
Tom Samplonius
Date:
On Sun, 31 Dec 2000, Ralf Mattes wrote:

> On Fri, Dec 29, 2000 at 07:05:41PM -0800, Tom Samplonius wrote:
> >   Also NFSv2 is limited to 2GB, even if the client and server have no
> > issues.  It is a protocol thing that is fixed in NFSv3.  I doubt that
> > anyone is putting their postgres databases on a NFS server, but you never
> > know.
>
> It would be a bad idea to put the database on NFS but the paragraph
> talks about possible problems during pg_dump and it doesn't seem
> too unreasonable to dump a database to an NFS-mounted area.

  Well NetApp has gone through quite a bit of work getting their NFS
filers Oracle and MS-SQL certified.  There are extensive tech notes on the
Netapp site about Oracle and MS-SQL on NFS servers.

> Ralf Mattes
>
> P.S: Happy new year :-)


Tom


Re: Rather incorrect text in admin guide

From
Bruce Momjian
Date:
I can only imagine how they got databases working through NFS.  Having
the backend on one server and the files on another is really quite risky.

>
> On Sun, 31 Dec 2000, Ralf Mattes wrote:
>
> > On Fri, Dec 29, 2000 at 07:05:41PM -0800, Tom Samplonius wrote:
> > >   Also NFSv2 is limited to 2GB, even if the client and server have no
> > > issues.  It is a protocol thing that is fixed in NFSv3.  I doubt that
> > > anyone is putting their postgres databases on a NFS server, but you never
> > > know.
> >
> > It would be a bad idea to put the database on NFS but the paragraph
> > talks about possible problems during pg_dump and it doesn't seem
> > too unreasonable to dump a database to an NFS-mounted area.
>
>   Well NetApp has gone through quite a bit of work getting their NFS
> filers Oracle and MS-SQL certified.  There are extensive tech notes on the
> Netapp site about Oracle and MS-SQL on NFS servers.
>
> > Ralf Mattes
> >
> > P.S: Happy new year :-)
>
>
> Tom
>
>


--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Re: Rather incorrect text in admin guide

From
Tom Samplonius
Date:
On Mon, 1 Jan 2001, Bruce Momjian wrote:

> I can only imagine how they got databases working through NFS.  Having
> the backend on one server and the files on another is really quite risky.

  Well, both Microsoft and Oracle support NFS mounted database device
files with NetApp filers.

Tom


Re: Rather incorrect text in admin guide

From
Bruce Momjian
Date:
>
> On Mon, 1 Jan 2001, Bruce Momjian wrote:
>
> > I can only imagine how they got databases working through NFS.  Having
> > the backend on one server and the files on another is really quite risky.
>
>   Well, both Microsoft and Oracle support NFS mounted database device
> files with NetApp filers.

Quite a trick.  NFS, being state-less, is really a bad platform for such
things.  I know there is NFS locking, but even that is not 100%, if I
remember correctly.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Re: Rather incorrect text in admin guide

From
The Hermit Hacker
Date:
On Tue, 2 Jan 2001, Bruce Momjian wrote:

> >
> > On Mon, 1 Jan 2001, Bruce Momjian wrote:
> >
> > > I can only imagine how they got databases working through NFS.  Having
> > > the backend on one server and the files on another is really quite risky.
> >
> >   Well, both Microsoft and Oracle support NFS mounted database device
> > files with NetApp filers.
>
> Quite a trick.  NFS, being state-less, is really a bad platform for such
> things.  I know there is NFS locking, but even that is not 100%, if I
> remember correctly.

NFS is state-less if you use UDP connections ... most modern Unices
support TCP NFS as well, providing you a stateful connection ...


Re: Rather incorrect text in admin guide

From
Michael Graff
Date:
Peter Eisentraut <peter_e@gmx.net> writes:

> > NetBSD (which I use) will allow file sizes up to 2^64 -- I don't think
> > anyone has generated a postgresql database that large yet.
>
> Possibly, but how is this relevant to what *could* be done?

Mainly that the whole section seems targeted at older OSs which cannot
write large files which pg_dumpall can generate.

Stating that this is only a requirement on some OSs seems to make
sense.

--Michael

Re: Rather incorrect text in admin guide

From
Ralf Mattes
Date:
On Fri, Dec 29, 2000 at 07:05:41PM -0800, Tom Samplonius wrote:
>   Also NFSv2 is limited to 2GB, even if the client and server have no
> issues.  It is a protocol thing that is fixed in NFSv3.  I doubt that
> anyone is putting their postgres databases on a NFS server, but you never
> know.

It would be a bad idea to put the database on NFS but the paragraph
talks about possible problems during pg_dump and it doesn't seem
too unreasonable to dump a database to an NFS-mounted area.

Ralf Mattes

P.S: Happy new year :-)


Re: Rather incorrect text in admin guide

From
Tom Samplonius
Date:
On Tue, 2 Jan 2001, The Hermit Hacker wrote:

> > > > I can only imagine how they got databases working through NFS.  Having
> > > > the backend on one server and the files on another is really quite risky.
> > >
> > >   Well, both Microsoft and Oracle support NFS mounted database device
> > > files with NetApp filers.
> >
> > Quite a trick.  NFS, being state-less, is really a bad platform for such
> > things.  I know there is NFS locking, but even that is not 100%, if I
> > remember correctly.
>
> NFS is state-less if you use UDP connections ... most modern Unices
> support TCP NFS as well, providing you a stateful connection ...

  A stateless procotol riding on a stateful protocol, is still stateless.
Think HTTP.

  The fact that NFS is stateless or not, does not matter for database
application.  In the supported configuration, only one server is allowed
to mount the database at one time anyway.  The only issue is making sure
that NFS writes actually get written when the NFS servers says they have
been written to ensure database consistancy.  Many NFS implementations ack
NFS writes before they are written NetApp uses a battery backed RAM as a
write cache.

Tom