Thread: Performance of ORDER BY

Performance of ORDER BY

From
Glenn Sullivan
Date:
I am wanting some ideas about improving the performance of ORDER BY in
our use.  I have a DB on the order of 500,000 rows and 50 columns.
The results are always sorted with ORDER BY.  Sometimes, the users end up
with a search that matches most of the rows.  In that case, I have a
LIMIT 5000 to keep the returned results under control.  However, the
sorting seems to take 10-60 sec.  If I do the same search without the
ORDER BY, it takes about a second.

I am currently on version 8.0.1 on Windows XP using a Dell Optiplex 280
with 1Gb of ram.  I have set sort_mem=100000 set.

Any ideas?

Thanks,
Glenn


Re: Performance of ORDER BY

From
"Luke Lonergan"
Date:
Glenn,

On 12/5/06 9:12 AM, "Glenn Sullivan" <glenn.sullivan@varianinc.com> wrote:

> I am wanting some ideas about improving the performance of ORDER BY in
> our use.  I have a DB on the order of 500,000 rows and 50 columns.
> The results are always sorted with ORDER BY.  Sometimes, the users end up
> with a search that matches most of the rows.  In that case, I have a
> LIMIT 5000 to keep the returned results under control.  However, the
> sorting seems to take 10-60 sec.  If I do the same search without the
> ORDER BY, it takes about a second.
>
> I am currently on version 8.0.1 on Windows XP using a Dell Optiplex 280
> with 1Gb of ram.  I have set sort_mem=100000 set.
>
> Any ideas?

Upgrade to 8.1 or 8.2, there were very large performance improvements to the
sort code made for 8.1+.  Also, when you've upgraded, you can experiment
with increasing work_mem to get better performance.  At some value of
work_mem (probably > 32MB) you will reach a plateau of performance, probably
4-5 times faster than what you see with 8.0.

- Luke



Re: Performance of ORDER BY

From
Tom Lane
Date:
Glenn Sullivan <glenn.sullivan@varianinc.com> writes:
> I am wanting some ideas about improving the performance of ORDER BY in
> our use.  I have a DB on the order of 500,000 rows and 50 columns.
> The results are always sorted with ORDER BY.  Sometimes, the users end up
> with a search that matches most of the rows.  In that case, I have a
> LIMIT 5000 to keep the returned results under control.  However, the
> sorting seems to take 10-60 sec.  If I do the same search without the
> ORDER BY, it takes about a second.

Does the ORDER BY match an index?  If so, is it using the index?
(See EXPLAIN.)

> I am currently on version 8.0.1 on Windows XP using a Dell Optiplex 280
> with 1Gb of ram.  I have set sort_mem=100000 set.

In 8.0 that might be counterproductively high --- we have seen cases
where more sort_mem = slower with the older sorting code.  I concur
with Luke's advice that you should update to 8.2 (not 8.1) to get the
improved sorting code.

            regards, tom lane

Re: Performance of ORDER BY

From
"Steinar H. Gunderson"
Date:
On Tue, Dec 05, 2006 at 01:02:06PM -0500, Tom Lane wrote:
> In 8.0 that might be counterproductively high --- we have seen cases
> where more sort_mem = slower with the older sorting code.  I concur
> with Luke's advice that you should update to 8.2 (not 8.1) to get the
> improved sorting code.

By the way, is the new sorting code any better for platforms that already
have a decent qsort() (like Linux)?

/* Steinar */
--
Homepage: http://www.sesse.net/

Re: Performance of ORDER BY

From
"A. Kretschmer"
Date:
am  Tue, dem 05.12.2006, um 13:02:06 -0500 mailte Tom Lane folgendes:
> In 8.0 that might be counterproductively high --- we have seen cases
> where more sort_mem = slower with the older sorting code.  I concur
> with Luke's advice that you should update to 8.2 (not 8.1) to get the
> improved sorting code.

Hey, yes, 8.2 is released! Great, and thanks to all the people, which
help to develop this great software.


Andreas
--
Andreas Kretschmer
Kontakt:  Heynitz: 035242/47215,   D1: 0160/7141639 (mehr: -> Header)
GnuPG-ID:   0x3FFF606C, privat 0x7F4584DA   http://wwwkeys.de.pgp.net

Re: Performance of ORDER BY

From
Tom Lane
Date:
"Steinar H. Gunderson" <sgunderson@bigfoot.com> writes:
> By the way, is the new sorting code any better for platforms that already
> have a decent qsort() (like Linux)?

It seemed better to us.  Linux' qsort() is really mergesort, which is
better sometimes but often worse --- mergesort tends to have a less
CPU-cache-friendly memory access distribution.  Another big problem with
the Linux version is that it pays no attention to sort_mem, but will
enthusiastically allocate lots of additional memory, thereby blowing
whatever cross-backend memory budgeting you might have been doing.

If you care there is quite a lot of discussion in the -hackers and
-performance archives from last spring or so.

            regards, tom lane

Re: Performance of ORDER BY

From
Stefan Kaltenbrunner
Date:
Steinar H. Gunderson wrote:
> On Tue, Dec 05, 2006 at 01:02:06PM -0500, Tom Lane wrote:
>> In 8.0 that might be counterproductively high --- we have seen cases
>> where more sort_mem = slower with the older sorting code.  I concur
>> with Luke's advice that you should update to 8.2 (not 8.1) to get the
>> improved sorting code.
>
> By the way, is the new sorting code any better for platforms that already
> have a decent qsort() (like Linux)?

yes - especially on-disk sorts will get some tremendous speedups in 8.2.


Stefan

Re: Performance of ORDER BY

From
Glenn Sullivan
Date:
Thanks to Luke and Tom for the input.  I guess this was good timing given that it looks like<br /> 8.2 was just
releasedtoday.   I will upgade to that before doing anything else.<br /><br /> Glenn<br /><br /> Tom Lane wrote:
<blockquotecite="mid7090.1165341726@sss.pgh.pa.us" type="cite"><pre wrap="">Glenn Sullivan <a
class="moz-txt-link-rfc2396E"href="mailto:glenn.sullivan@varianinc.com"><glenn.sullivan@varianinc.com></a>
writes:</pre><blockquote type="cite"><pre wrap="">I am wanting some ideas about improving the performance of ORDER BY
in
our use.  I have a DB on the order of 500,000 rows and 50 columns.
The results are always sorted with ORDER BY.  Sometimes, the users end up
with a search that matches most of the rows.  In that case, I have a
LIMIT 5000 to keep the returned results under control.  However, the
sorting seems to take 10-60 sec.  If I do the same search without the
ORDER BY, it takes about a second.     </pre></blockquote><pre wrap="">
Does the ORDER BY match an index?  If so, is it using the index?
(See EXPLAIN.)
 </pre><blockquote type="cite"><pre wrap="">I am currently on version 8.0.1 on Windows XP using a Dell Optiplex 280
with 1Gb of ram.  I have set sort_mem=100000 set.   </pre></blockquote><pre wrap="">
In 8.0 that might be counterproductively high --- we have seen cases
where more sort_mem = slower with the older sorting code.  I concur
with Luke's advice that you should update to 8.2 (not 8.1) to get the
improved sorting code.
        regards, tom lane
 </pre></blockquote>