Thread: Re: [HACKERS] tables >2GB

> > So, how about we fix the storage manager instead?
>
> I don't recall, what exactly breaks when going over 2 gig? I don't have the
> disk space available, otherwise I'd debug this.  I can still try if I knew
> what the problem was...this code isn't all that complex.

I agree.

> OK...here is a patch that will cause the magnetic disk storage manager to
> not try to split files in 2 gig chunks.  It will just try to get another
> block.
>
> If applied, everything is just as before. But if LET_OS_MANAGE_FILESIZE
> is defined, the chaining disappears and the file just keeps on going,
> and going, and going, til the OS barfs.
>
> Are there #defines in the system includes that could be used to determine
> a max file size?  If so, then I'd think that this would be something
> to add to configure.  If files over 2 gig are not allowed, then the old
> code would compile.
>
> Anyway, if the patch looks ok to the powers-that-be or if there is some
> thing else to be changed, let me know and I'll resubmit it to PATCHES.
>
> Compiled and regressed ok with and without LET_OS_MANAGE_FILESIZE, but
> then again there aren't any regression tables over 2 gig. :)
>

Well, BSD/OS goes over 2gig, but the postgreSQL code uses lseek, which
returns long, so even though I can handle larger files, the lseek()
can't because long is 32-bits.  Looks like only Alpha can handle those
files based on our current code.

Thanks to our success, this looks like something we will have to deal
with.


--
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

* Bruce Momjian
|
| Well, BSD/OS goes over 2gig, but the postgreSQL code uses lseek, which
| returns long, so even though I can handle larger files, the lseek()
| can't because long is 32-bits.

Are you sure?  In NetBSD, lseek() is declared to return an off_t,
which again is defined to be a 64bit quantity.  I would assume that
BSD/OS did it the same way -- in fact, I'd be surprised if not.

-tih
--
Popularity is the hallmark of mediocrity.  --Niles Crane, "Frasier"

>
> * Bruce Momjian
> |
> | Well, BSD/OS goes over 2gig, but the postgreSQL code uses lseek, which
> | returns long, so even though I can handle larger files, the lseek()
> | can't because long is 32-bits.
>
> Are you sure?  In NetBSD, lseek() is declared to return an off_t,
> which again is defined to be a 64bit quantity.  I would assume that
> BSD/OS did it the same way -- in fact, I'd be surprised if not.

Oops, you are right:

    typedef quad_t          off_t;

I thought they added fgetpos() only for 64-bit quantities, and did not
change the return value of lseek.  However:

    sys/types.h:76: typedef quad_t          off_t;          /* file offset*/

so you are right, but our code:

    fd.c:110:       long            seekPos;
    fd.c:263:       fileP->seekPos = (long) lseek(fileP->fd, 0L, SEEK_CUR);

so it still will not work because the code is not defining seekPos as
off_t.  We need to get this code cleaned up/fixed.

How could they make such a mistake and assume it is a long, unless this
thing gets passed around in the backend, and they don't want to
reference off_t all over the place?  That code needs cleanup.

--
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

* Bruce Momjian
|
| How could they make such a mistake and assume it is a long, [...]

"All the world's a VAX!"  Getting rid of (and tidying up after)
explicit "long lseek();" declarations is a major part of porting
old software to new UNIXes.

-tih
--
Popularity is the hallmark of mediocrity.  --Niles Crane, "Frasier"

>
> * Bruce Momjian
> |
> | How could they make such a mistake and assume it is a long, [...]
>
> "All the world's a VAX!"  Getting rid of (and tidying up after)
> explicit "long lseek();" declarations is a major part of porting
> old software to new UNIXes.

It's on our TODO list now.

--
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)