Re: Performance issues when the number of records are around 10 Million - Mailing list pgsql-performance

From Kevin Grittner
Subject Re: Performance issues when the number of records are around 10 Million
Date
Msg-id 4BEA50DE02000025000315EF@gw.wicourts.gov
Whole thread Raw
Responses Re: Performance issues when the number of records are around 10 Million  (Craig James <craig_james@emolecules.com>)
Re: Performance issues when the number of records are around 10 Million  (venu madhav <venutaurus539@gmail.com>)
List pgsql-performance
venu madhav  wrote:

>>> AND e.timestamp >= '1270449180'
>>> AND e.timestamp < '1273473180'
>>> ORDER BY.
>>> e.cid DESC,
>>> e.cid DESC
>>> limit 21
>>> offset 10539780

> The second column acts as a secondary key for sorting if the
> primary sorting key is a different column. For this query both of
> them are same.

Any chance you could just leave the second one off in that case?

> This query is part of an application which allows user to select
> time ranges and retrieve the data in that interval. Hence the time
> stamp.

Which, of course, is going to affect the number of rows.  Which
leaves me wondering how you know that once you select and sequence
the result set you need to read past and ignore exactly 10539780
rows to get to the last page.

> To have it in some particular order we're doing order by.

Which will affect which rows are at any particular offset.

> If the records are more in the interval,

How do you know that before you run your query?

> we display in sets of 20/30 etc. The user also has the option to
> browse through any of those records hence the limit and offset.

Have you considered alternative techniques for paging?  You might
use values at the edges of the page to run a small query (limit, no
offset) when they page.  You might generate all the pages on the
first pass and cache them for a while.

> When the user asks for the last set of 20 records, this query gets
> executed.

The DESC on the ORDER BY makes it look like you're trying to use the
ORDER BY to get to the end, but then your offset tells PostgreSQL to
skip the 10.5 million result rows with the highest keys.  Is the
"last page" the one with the highest or lowest values for cid?

-Kevin



pgsql-performance by date:

Previous
From: Josh Berkus
Date:
Subject: Re: Performance issues when the number of records are around 10 Million
Next
From: "Kevin Grittner"
Date:
Subject: Re: Performance issues when the number of records are around 10 Million