Thread: tables >2GB
Now that we know the storage manager code that splits tables over 2GB into separate files doesn't work(Irix), can we rip out that code and just use the OS code to access >2GB files as normal files. Now, most OS's can support 64-bit files and file sizes. Because it is isolated in the storage manager, it should be easy. -- Bruce Momjian maillist@candle.pha.pa.us
Bruce Momjian wrote: > > Now that we know the storage manager code that splits tables over 2GB > into separate files doesn't work(Irix), can we rip out that code and > just use the OS code to access >2GB files as normal files. Now, most > OS's can support 64-bit files and file sizes. > > Because it is isolated in the storage manager, it should be easy. Someday we'll get TABLESPACEs and fixed multi-chunk code could allow to store chunks in different TABLESPACEs created on _different disks_ - imho, ability to store a table on > 1 disk is good thing. And so, I would suggest just add elog(ERROR) to mdextend() now, with recommendation to increase RELSEG_SIZE... Vadim
> > Now that we know the storage manager code that splits tables over 2GB > into separate files doesn't work(Irix), can we rip out that code and > just use the OS code to access >2GB files as normal files. Now, most > OS's can support 64-bit files and file sizes. > > Because it is isolated in the storage manager, it should be easy. Can someone knowledgeable make a patch for this for our mega-patch? -- Bruce Momjian | 830 Blythe Avenue maillist@candle.pha.pa.us | Drexel Hill, Pennsylvania 19026 + If your life is a hard drive, | (610) 353-9879(w) + Christ can be your backup. | (610) 853-3000(h)
On Thu, 19 Mar 1998, Bruce Momjian wrote:
> >
> > Now that we know the storage manager code that splits tables over 2GB
> > into separate files doesn't work(Irix), can we rip out that code and
> > just use the OS code to access >2GB files as normal files. Now, most
> > OS's can support 64-bit files and file sizes.
> >
> > Because it is isolated in the storage manager, it should be easy.
>
> Can someone knowledgeable make a patch for this for our mega-patch?
*Only* if its in before 9am AST (or is it ADT?) on Friday
morning...:)
> > Now that we know the storage manager code that splits tables over 2GB
> > into separate files doesn't work(Irix), can we rip out that code and
> > just use the OS code to access >2GB files as normal files. Now, most
> > OS's can support 64-bit files and file sizes.
> >
> > Because it is isolated in the storage manager, it should be easy.
>
> Can someone knowledgeable make a patch for this for our mega-patch?
>
There are still quite a few OS's out there that do not support >2GB files
yet. Even my beloved Linux (x86)...
So, how about we fix the storage manager instead?
A neat thing that Illustra does is allow you to stripe a table across
multiple directories. You get big tables, easy storage management, and
a nice performance boost.
create stripedir('stripe1', '/disk1/data/stripe1');
create stripedir('stripe2', '/disk2/data/stripe2');
create table giant_table (...) with (stripes 4, 'stripe1', 'stripe2');
-- the '4' is the number of pages to interleave.
Then the smgr just distributes the blocks alternately across the stripes.
read_block(blockno, ...stripeinfo)
{
...
stripe = (blockno / stripe_interleave ) % number_of_stripes;
stripe_block = blockno / number_of_stripes;
fd = stripe_info->fd[stripe];
lseek(fd, stripe_block * BLOCKSIZE, SEEK_SET);
...
}
All vastly oversimplified of course....
-dg
David Gould dg@illustra.com 510.628.3783 or 510.305.9468
Informix Software (No, really) 300 Lakeside Drive Oakland, CA 94612
- I realize now that irony has no place in business communications.
Does this act as a notice of volunteering to work on this aspect of the
code? *Grin*
On Thu, 19 Mar 1998, David Gould wrote:
> > > Now that we know the storage manager code that splits tables over 2GB
> > > into separate files doesn't work(Irix), can we rip out that code and
> > > just use the OS code to access >2GB files as normal files. Now, most
> > > OS's can support 64-bit files and file sizes.
> > >
> > > Because it is isolated in the storage manager, it should be easy.
> >
> > Can someone knowledgeable make a patch for this for our mega-patch?
> >
>
> There are still quite a few OS's out there that do not support >2GB files
> yet. Even my beloved Linux (x86)...
>
> So, how about we fix the storage manager instead?
>
> A neat thing that Illustra does is allow you to stripe a table across
> multiple directories. You get big tables, easy storage management, and
> a nice performance boost.
>
> create stripedir('stripe1', '/disk1/data/stripe1');
> create stripedir('stripe2', '/disk2/data/stripe2');
>
> create table giant_table (...) with (stripes 4, 'stripe1', 'stripe2');
> -- the '4' is the number of pages to interleave.
>
> Then the smgr just distributes the blocks alternately across the stripes.
>
> read_block(blockno, ...stripeinfo)
> {
> ...
> stripe = (blockno / stripe_interleave ) % number_of_stripes;
> stripe_block = blockno / number_of_stripes;
>
> fd = stripe_info->fd[stripe];
> lseek(fd, stripe_block * BLOCKSIZE, SEEK_SET);
> ...
> }
>
> All vastly oversimplified of course....
>
> -dg
>
> David Gould dg@illustra.com 510.628.3783 or 510.305.9468
> Informix Software (No, really) 300 Lakeside Drive Oakland, CA 94612
> - I realize now that irony has no place in business communications.
>
>
>
>
Bruce Momjian wrote: > > > > > Now that we know the storage manager code that splits tables over 2GB > > into separate files doesn't work(Irix), can we rip out that code and > > just use the OS code to access >2GB files as normal files. Now, most > > OS's can support 64-bit files and file sizes. > > > > Because it is isolated in the storage manager, it should be easy. > > Can someone knowledgeable make a patch for this for our mega-patch? But, could it not be useful to be able to use multiple files per table? Suppose someone wants to spread them out on different disks to increase access performance? And what about tables over 2^64 bytes size? There will never be disks of that size? Now, remember what people said about 2^32 byte files, and years after 1999, and 64k RAM, and about all inventions already being invented, and... :) /* m */