Thread: Performance of ORDER BY
I am wanting some ideas about improving the performance of ORDER BY in our use. I have a DB on the order of 500,000 rows and 50 columns. The results are always sorted with ORDER BY. Sometimes, the users end up with a search that matches most of the rows. In that case, I have a LIMIT 5000 to keep the returned results under control. However, the sorting seems to take 10-60 sec. If I do the same search without the ORDER BY, it takes about a second. I am currently on version 8.0.1 on Windows XP using a Dell Optiplex 280 with 1Gb of ram. I have set sort_mem=100000 set. Any ideas? Thanks, Glenn
Glenn, On 12/5/06 9:12 AM, "Glenn Sullivan" <glenn.sullivan@varianinc.com> wrote: > I am wanting some ideas about improving the performance of ORDER BY in > our use. I have a DB on the order of 500,000 rows and 50 columns. > The results are always sorted with ORDER BY. Sometimes, the users end up > with a search that matches most of the rows. In that case, I have a > LIMIT 5000 to keep the returned results under control. However, the > sorting seems to take 10-60 sec. If I do the same search without the > ORDER BY, it takes about a second. > > I am currently on version 8.0.1 on Windows XP using a Dell Optiplex 280 > with 1Gb of ram. I have set sort_mem=100000 set. > > Any ideas? Upgrade to 8.1 or 8.2, there were very large performance improvements to the sort code made for 8.1+. Also, when you've upgraded, you can experiment with increasing work_mem to get better performance. At some value of work_mem (probably > 32MB) you will reach a plateau of performance, probably 4-5 times faster than what you see with 8.0. - Luke
Glenn Sullivan <glenn.sullivan@varianinc.com> writes:
> I am wanting some ideas about improving the performance of ORDER BY in
> our use. I have a DB on the order of 500,000 rows and 50 columns.
> The results are always sorted with ORDER BY. Sometimes, the users end up
> with a search that matches most of the rows. In that case, I have a
> LIMIT 5000 to keep the returned results under control. However, the
> sorting seems to take 10-60 sec. If I do the same search without the
> ORDER BY, it takes about a second.
Does the ORDER BY match an index? If so, is it using the index?
(See EXPLAIN.)
> I am currently on version 8.0.1 on Windows XP using a Dell Optiplex 280
> with 1Gb of ram. I have set sort_mem=100000 set.
In 8.0 that might be counterproductively high --- we have seen cases
where more sort_mem = slower with the older sorting code. I concur
with Luke's advice that you should update to 8.2 (not 8.1) to get the
improved sorting code.
regards, tom lane
On Tue, Dec 05, 2006 at 01:02:06PM -0500, Tom Lane wrote: > In 8.0 that might be counterproductively high --- we have seen cases > where more sort_mem = slower with the older sorting code. I concur > with Luke's advice that you should update to 8.2 (not 8.1) to get the > improved sorting code. By the way, is the new sorting code any better for platforms that already have a decent qsort() (like Linux)? /* Steinar */ -- Homepage: http://www.sesse.net/
am Tue, dem 05.12.2006, um 13:02:06 -0500 mailte Tom Lane folgendes: > In 8.0 that might be counterproductively high --- we have seen cases > where more sort_mem = slower with the older sorting code. I concur > with Luke's advice that you should update to 8.2 (not 8.1) to get the > improved sorting code. Hey, yes, 8.2 is released! Great, and thanks to all the people, which help to develop this great software. Andreas -- Andreas Kretschmer Kontakt: Heynitz: 035242/47215, D1: 0160/7141639 (mehr: -> Header) GnuPG-ID: 0x3FFF606C, privat 0x7F4584DA http://wwwkeys.de.pgp.net
"Steinar H. Gunderson" <sgunderson@bigfoot.com> writes:
> By the way, is the new sorting code any better for platforms that already
> have a decent qsort() (like Linux)?
It seemed better to us. Linux' qsort() is really mergesort, which is
better sometimes but often worse --- mergesort tends to have a less
CPU-cache-friendly memory access distribution. Another big problem with
the Linux version is that it pays no attention to sort_mem, but will
enthusiastically allocate lots of additional memory, thereby blowing
whatever cross-backend memory budgeting you might have been doing.
If you care there is quite a lot of discussion in the -hackers and
-performance archives from last spring or so.
regards, tom lane
Steinar H. Gunderson wrote: > On Tue, Dec 05, 2006 at 01:02:06PM -0500, Tom Lane wrote: >> In 8.0 that might be counterproductively high --- we have seen cases >> where more sort_mem = slower with the older sorting code. I concur >> with Luke's advice that you should update to 8.2 (not 8.1) to get the >> improved sorting code. > > By the way, is the new sorting code any better for platforms that already > have a decent qsort() (like Linux)? yes - especially on-disk sorts will get some tremendous speedups in 8.2. Stefan
Thanks to Luke and Tom for the input. I guess this was good timing given that it looks like<br /> 8.2 was just
releasedtoday. I will upgade to that before doing anything else.<br /><br /> Glenn<br /><br /> Tom Lane wrote:
<blockquotecite="mid7090.1165341726@sss.pgh.pa.us" type="cite"><pre wrap="">Glenn Sullivan <a
class="moz-txt-link-rfc2396E"href="mailto:glenn.sullivan@varianinc.com"><glenn.sullivan@varianinc.com></a>
writes:</pre><blockquote type="cite"><pre wrap="">I am wanting some ideas about improving the performance of ORDER BY
in
our use. I have a DB on the order of 500,000 rows and 50 columns.
The results are always sorted with ORDER BY. Sometimes, the users end up
with a search that matches most of the rows. In that case, I have a
LIMIT 5000 to keep the returned results under control. However, the
sorting seems to take 10-60 sec. If I do the same search without the
ORDER BY, it takes about a second. </pre></blockquote><pre wrap="">
Does the ORDER BY match an index? If so, is it using the index?
(See EXPLAIN.)
</pre><blockquote type="cite"><pre wrap="">I am currently on version 8.0.1 on Windows XP using a Dell Optiplex 280
with 1Gb of ram. I have set sort_mem=100000 set. </pre></blockquote><pre wrap="">
In 8.0 that might be counterproductively high --- we have seen cases
where more sort_mem = slower with the older sorting code. I concur
with Luke's advice that you should update to 8.2 (not 8.1) to get the
improved sorting code.
regards, tom lane
</pre></blockquote>