Thread: I've got it, now should I commit it?
After reading a couple more complaints of hashtable-overflow error
messages, I went ahead and rewrote the hash join modules so that they
don't use fixed-size hash buckets and a fixed-size overflow area.
Instead, each bucket is just a linked list of tuples (thus no wasted
space for underused buckets) and everything is put into a private portal
so that reclaiming the space is easy/quick. The code is noticeably
shorter and more readable than before.
The limited amount of testing I've been able to do here shows no
problems.
Now: do I commit it, or wait till after 6.5? I promised Marc the latter
a couple weeks ago, but I am mighty tempted to just go for it...
regards, tom lane
> After reading a couple more complaints of hashtable-overflow error > messages, I went ahead and rewrote the hash join modules so that they > don't use fixed-size hash buckets and a fixed-size overflow area. > Instead, each bucket is just a linked list of tuples (thus no wasted > space for underused buckets) and everything is put into a private portal > so that reclaiming the space is easy/quick. The code is noticeably > shorter and more readable than before. > > The limited amount of testing I've been able to do here shows no > problems. > > Now: do I commit it, or wait till after 6.5? I promised Marc the latter > a couple weeks ago, but I am mighty tempted to just go for it... Shhh. He will never know. Did you promise Marc, or did you answer him evasively, like I suggested? Basically, with the new optimizer, this may be a bug fix because of the more frequent hashjoins. That has always been my smokescreen to add the feature. :-) -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
On Tue, 18 May 1999, Bruce Momjian wrote:
> > After reading a couple more complaints of hashtable-overflow error
> > messages, I went ahead and rewrote the hash join modules so that they
> > don't use fixed-size hash buckets and a fixed-size overflow area.
> > Instead, each bucket is just a linked list of tuples (thus no wasted
> > space for underused buckets) and everything is put into a private portal
> > so that reclaiming the space is easy/quick. The code is noticeably
> > shorter and more readable than before.
> >
> > The limited amount of testing I've been able to do here shows no
> > problems.
> >
> > Now: do I commit it, or wait till after 6.5? I promised Marc the latter
> > a couple weeks ago, but I am mighty tempted to just go for it...
>
> Shhh. He will never know. Did you promise Marc, or did you answer him
> evasively, like I suggested?
>
> Basically, with the new optimizer, this may be a bug fix because of the
> more frequent hashjoins. That has always been my smokescreen to add the
> feature.
Tom...make you a deal. If you are confident enough with the code that
when v6.5 goes out in ~13days, it won't generate more bug reports then its
fixing...go for it. :)
Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org
> > Basically, with the new optimizer, this may be a bug fix because of the > > more frequent hashjoins. That has always been my smokescreen to add the > > feature. > > Tom...make you a deal. If you are confident enough with the code that > when v6.5 goes out in ~13days, it won't generate more bug reports then its > fixing...go for it. :) > Ah, man, that was way too easy. :-) -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
The Hermit Hacker <scrappy@hub.org> writes:
>> Basically, with the new optimizer, this may be a bug fix because of the
>> more frequent hashjoins. That has always been my smokescreen to add the
>> feature.
> Tom...make you a deal. If you are confident enough with the code that
> when v6.5 goes out in ~13days, it won't generate more bug reports then its
> fixing...go for it. :)
OK, you're on --- I feel pretty good about this code, although I'm never
prepared to guarantee zero bugs ;-). If there are any, we can hope
they'll show up before the end of beta.
A note for anyone testing the new code: the hashtable size (which is now
a target estimate, not a hard limit) is now driven by the postmaster's
-S switch, not the -B switch. -S seems more reasonable since the table
is private memory in a backend, not shared memory.
regards, tom lane
> The Hermit Hacker <scrappy@hub.org> writes: > >> Basically, with the new optimizer, this may be a bug fix because of the > >> more frequent hashjoins. That has always been my smokescreen to add the > >> feature. > > > Tom...make you a deal. If you are confident enough with the code that > > when v6.5 goes out in ~13days, it won't generate more bug reports then its > > fixing...go for it. :) > > OK, you're on --- I feel pretty good about this code, although I'm never > prepared to guarantee zero bugs ;-). If there are any, we can hope > they'll show up before the end of beta. > > A note for anyone testing the new code: the hashtable size (which is now > a target estimate, not a hard limit) is now driven by the postmaster's > -S switch, not the -B switch. -S seems more reasonable since the table > is private memory in a backend, not shared memory. I see no documenation that -B was ever used for hash size. I see -B for shared buffers for both postmaster and postgres manual pages. -- Bruce Momjian | http://www.op.net/~candle maillist@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
Bruce Momjian <maillist@candle.pha.pa.us> writes:
>> A note for anyone testing the new code: the hashtable size (which is now
>> a target estimate, not a hard limit) is now driven by the postmaster's
>> -S switch, not the -B switch.
> I see no documenation that -B was ever used for hash size.
Er, did I say anything about documentation?
The code *was* using NBuffers to size the hashtable, whether or not
that was ever documented anywhere except in the "hash table out of
memory. Use -B parameter to increase buffers" message. Now it uses
the SortMem variable.
I do have it on my to-do list to update the relevant documentation.
(Yo, Thomas: what's the deadline for 6.5 doco changes? I've got a bunch
of doc to-dos that I suspect I'd better get moving on...)
regards, tom lane
> (Yo, Thomas: what's the deadline for 6.5 doco changes? I've got a bunch
> of doc to-dos that I suspect I'd better get moving on...)
Yup. Nominally, I should have frozen about May 15, but I still have
some writing I want to do. I can freeze docs in stages; which docs are
you planning on touching?
Bruce, can we get the sgml version of release notes started?
Vadim, you had mentioned some docs for MVCC; where would that show up?
If nothing else, we should update ref/lock.sgml and ref/set.sgml to
cover the grammar changes. And it would be great to have some words
for the User's Guide.
- Thomas
--
Thomas Lockhart lockhart@alumni.caltech.edu
South Pasadena, California
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
> I can freeze docs in stages; which docs are
> you planning on touching?
Several, but for most of them I have only small changes. I will try to
do those tonight so that as much as possible can be frozen, and then let
you know what I still have to work on.
Is there any equivalent in the SGML docs to the postmaster.1 and
postgres.1 man pages (specifically, doco for the postmaster/backend
command line switches)? I couldn't find it but maybe it's there...
regards, tom lane
> Several, but for most of them I have only small changes. I will try to
> do those tonight so that as much as possible can be frozen, and then let
> you know what I still have to work on.
Great.
> Is there any equivalent in the SGML docs to the postmaster.1 and
> postgres.1 man pages (specifically, doco for the postmaster/backend
> command line switches)? I couldn't find it but maybe it's there...
No, but I want to add them by converting the man pages to User's Guide
reference pages. Will wait a day or two for you to update the man
pages, unless you would prefer to have me convert and then make your
changes directly in sgml.
I'm doing a small reorg on the Admin Guide. Most chapters stay the
same, but things are streamlined and flow better.
- Thomas
--
Thomas Lockhart lockhart@alumni.caltech.edu
South Pasadena, California
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
>> Is there any equivalent in the SGML docs to the postmaster.1 and
>> postgres.1 man pages (specifically, doco for the postmaster/backend
>> command line switches)? I couldn't find it but maybe it's there...
> No, but I want to add them by converting the man pages to User's Guide
> reference pages. Will wait a day or two for you to update the man
> pages,
Already committed my changes to the .1 files; you may fire when ready.
(Unless anyone else has updates to make there?)
regards, tom lane