Re: [PERFORM] Bad n_distinct estimation; hacks suggested? - Mailing list pgsql-hackers

From Greg Stark
Subject Re: [PERFORM] Bad n_distinct estimation; hacks suggested?
Date
Msg-id 87u0lt5efo.fsf@stark.xeocode.com
Whole thread Raw
Responses Re: [PERFORM] Bad n_distinct estimation; hacks suggested?
List pgsql-hackers
This one looks *really* good. 
http://www.aladdin.cs.cmu.edu/papers/pdfs/y2001/dist_sampl.pdf

It does require a single full table scan but it works in O(n) time and
constant space and it guarantees the confidence intervals for the estimates it
provides like the histograms do for regular range scans.

It can even keep enough data to provide estimates for n_distinct when
unrelated predicates are applied. I'm not sure Postgres would want to do this
though; this seems like it's part of the cross-column correlation story more
than the n_distinct story. It seems to require keeping an entire copy of the
sampled record in the stats tables which would be prohibitive quickly in wide
tables (it would be O(n^2) storage in the number of columns) .

It also seems like a lot of work to implement. Nothing particular that would
be impossible, but it does require storing a moderately complex data
structure. Perhaps Postgres's new support for data structures will make this
easier.

-- 
greg



pgsql-hackers by date:

Previous
From: David Wheeler
Date:
Subject: Re: DO INSTEAD and conditional rules
Next
From: Greg Stark
Date:
Subject: Re: [PERFORM] Bad n_distinct estimation; hacks suggested?