Thread: Same query doing slow then quick

Hi,

I'm new here so i hope i don't do mistakes.

I'm having a serious performance issue in postgresql.

I have tables containing adresses with X,Y GPS coordinates and tables with
zoning and square of gps coordinates.

Basicly it looks like

adresses_01 (id,X,Y)
gps_01 (id,x_min,x_max,y_min,y_max).

[code]
"
SELECT
    t2.id,
FROM
    tables_gps.gps_01 t1
INNER JOIN
    tables_adresses.adresses_01 t2
ON
    t2."X" BETWEEN t1.x_min AND t1.x_max AND t2."Y" BETWEEN t1.y_min AND
t1.y_max
WHERE
    t2.id='0'
"
[/code]

I have something like 250000rows in each table.

Now when i execute this on adresses_01 and gps_01, the request complete in a
few minutes.
But when doing it on adresses_02 and gps_02 (same number of rows
approximately) the query takes 5hours.

I have indexes on adresses on X,Y and an index in gps on
x_min,y_min,x_max,y_max.

Now i do updates in result of this query on ID (so i have an index on ID
too).

My question is ... Why ? (;o). And also, do i need to use CLUSTER (i don't
really understand what it does). And if so. Do i need to CLUSTER the id ? Or
the X,Y index ?

It may be not really clear so just ask questions if you don't get when i
mean or if you need specs or anything. I just moved from MySql to PostgreSql
last month.

Thanks in advance :)



--
View this message in context: http://postgresql.1045698.n5.nabble.com/Same-query-doing-slow-then-quick-tp5725486.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.

Here is the answer to Ray Stell who send me the wiki page of Slow Query. I
hope i detailed all you wanted (i basicly pasted the page and add my
answers).

Full Table and Index Schema:

schema tables_adresses
"Tables"
tables_adresses.adresses_XX (id (serial), X(Double precision),Y (Double
precision)).
"Indexes"
adresses_XX_pkey (Primary key, btree)
calcul_XX (non unique, Btree on X,Y)

schema tables_gps
"Tables"
tables_gps.gps_XX (id (int),x_max(numeric(10,5)), y_max
(numeric(10,5)),x_min(numeric(10,5)),y_min(numeric(10,5)))
"Indexes"
calculs_XX (non unique Btree x_min,x_max,y_min,y_max)
gps_10_pkey (Primary key on id btree)

Approximate rows 250000.
No large objects in it (just data)
No NULL
receives a large number of UPDATEs or DELETEs regularly
is growing daily

I can't post an EXPLAIN ANALYZE because of the 6hour query time.

Postgres version: 9.1

History: was this query always slow, : "YES"

Hardware: Ubuntu server last version 32bits

Daily VACUUM FULL ANALYZE, REINDEX TABLE on all the tables.

WAL Configuration: Whats a WAL ?

GUC Settings: i didn't change anything. All is standard.

shared_buffers should be 10% to 25% of available RAM (it's on 24MB and can't
go higher. The server has 4Gb)

effective_cache_size should be 75% of available RAM => I don't now what this
is.

Test changing work_mem: increase it to 8MB, 32MB, 256MB, 1GB. Does it make a
difference? "No"




--
View this message in context:
http://postgresql.1045698.n5.nabble.com/Same-query-doing-slow-then-quick-tp5725486p5725491.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.

On 09/26/2012 15:03, FFW_Rude wrote:
> Here is the answer to Ray Stell who send me the wiki page of Slow Query. I
> hope i detailed all you wanted (i basicly pasted the page and add my
> answers).
>
> Full Table and Index Schema:
>
> schema tables_adresses
> "Tables"
> tables_adresses.adresses_XX (id (serial), X(Double precision),Y (Double
> precision)).
> "Indexes"
> adresses_XX_pkey (Primary key, btree)
> calcul_XX (non unique, Btree on X,Y)
>
> schema tables_gps
> "Tables"
> tables_gps.gps_XX (id (int),x_max(numeric(10,5)), y_max
> (numeric(10,5)),x_min(numeric(10,5)),y_min(numeric(10,5)))
> "Indexes"
> calculs_XX (non unique Btree x_min,x_max,y_min,y_max)
> gps_10_pkey (Primary key on id btree)
>
> Approximate rows 250000.
> No large objects in it (just data)
> No NULL
> receives a large number of UPDATEs or DELETEs regularly
> is growing daily
>
> I can't post an EXPLAIN ANALYZE because of the 6hour query time.
>
> Postgres version: 9.1
>
> History: was this query always slow, : "YES"
>
> Hardware: Ubuntu server last version 32bits
>
> Daily VACUUM FULL ANALYZE, REINDEX TABLE on all the tables.
>
> WAL Configuration: Whats a WAL ?
>
> GUC Settings: i didn't change anything. All is standard.
>
> shared_buffers should be 10% to 25% of available RAM (it's on 24MB and can't
> go higher. The server has 4Gb)
>
> effective_cache_size should be 75% of available RAM =>  I don't now what this
> is.

before looking further, please configure shared_buffers and
effective_cache_size properly, it's fundamental
you'll probably need to raise SHMALL/SHMMAX, take a look at:
http://www.postgresql.org/docs/current/static/kernel-resources.html
for 4GB of RAM I start with shared_buffers to 512MB and
effective_cache_size to 2GB

> Test changing work_mem: increase it to 8MB, 32MB, 256MB, 1GB. Does it make a
> difference? "No"

default work_mem is very small, set it to something like 16MB

>
>
>
> --
> View this message in context:
http://postgresql.1045698.n5.nabble.com/Same-query-doing-slow-then-quick-tp5725486p5725491.html
> Sent from the PostgreSQL - performance mailing list archive at Nabble.com.
>
>


--
No trees were killed in the creation of this message.
However, many electrons were terribly inconvenienced.

Hi,

Thank you for your answer.

It was already at 16MB and i upped it just this morning to 64MB. Still no change

Rude - Last Territory

Ou écouter ?

http://www.deezer.com/fr/music/last-territory/the-last-hope-3617781       (Post-apocalyptic Metal)

http://www.deezer.com/fr/music/rude-undertaker    (Pop-Rock)

Ou acheter ?

La Fnac

http://recherche.fnac.com/fmia14622213/Last-Territory

http://recherche.fnac.com/fmia14770622/Rude-Undertaker

iTunes

http://itunes.apple.com/us/artist/last-territory/id533857009?ign-mpt=uo%3D4

---

Date: Wed, 26 Sep 2012 06:22:35 -0700
From: [hidden email]
To: [hidden email]
Subject: Re: Same query doing slow then quick

On 09/26/2012 15:03, FFW_Rude wrote:

> Here is the answer to Ray Stell who send me the wiki page of Slow Query. I
> hope i detailed all you wanted (i basicly pasted the page and add my
> answers).
>
> Full Table and Index Schema:
>
> schema tables_adresses
> "Tables"
> tables_adresses.adresses_XX (id (serial), X(Double precision),Y (Double
> precision)).
> "Indexes"
> adresses_XX_pkey (Primary key, btree)
> calcul_XX (non unique, Btree on X,Y)
>
> schema tables_gps
> "Tables"
> tables_gps.gps_XX (id (int),x_max(numeric(10,5)), y_max
> (numeric(10,5)),x_min(numeric(10,5)),y_min(numeric(10,5)))
> "Indexes"
> calculs_XX (non unique Btree x_min,x_max,y_min,y_max)
> gps_10_pkey (Primary key on id btree)
>
> Approximate rows 250000.
> No large objects in it (just data)
> No NULL
> receives a large number of UPDATEs or DELETEs regularly
> is growing daily
>
> I can't post an EXPLAIN ANALYZE because of the 6hour query time.
>
> Postgres version: 9.1
>
> History: was this query always slow, : "YES"
>
> Hardware: Ubuntu server last version 32bits
>
> Daily VACUUM FULL ANALYZE, REINDEX TABLE on all the tables.
>
> WAL Configuration: Whats a WAL ?
>
> GUC Settings: i didn't change anything. All is standard.
>
> shared_buffers should be 10% to 25% of available RAM (it's on 24MB and can't
> go higher. The server has 4Gb)
>
> effective_cache_size should be 75% of available RAM =>  I don't now what this
> is.

before looking further, please configure shared_buffers and

effective_cache_size properly, it's fundamental
you'll probably need to raise SHMALL/SHMMAX, take a look at:
http://www.postgresql.org/docs/current/static/kernel-resources.html
for 4GB of RAM I start with shared_buffers to 512MB and
effective_cache_size to 2GB

> Test changing work_mem: increase it to 8MB, 32MB, 256MB, 1GB. Does it make a
> difference? "No"

default work_mem is very small, set it to something like 16MB

>
>
>
> --
> View this message in context: http://postgresql.1045698.n5.nabble.com/Same-query-doing-slow-then-quick-tp5725486p5725491.html
> Sent from the PostgreSQL - performance mailing list archive at Nabble.com.
>
>


--
No trees were killed in the creation of this message.
However, many electrons were terribly inconvenienced.



--
Sent via pgsql-performance mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

jcigar.vcf (304 bytes) Download Attachment

---

If you reply to this email, your message will be added to the discussion below:

http://postgresql.1045698.n5.nabble.com/Same-query-doing-slow-then-quick-tp5725486p5725493.html

To unsubscribe from Same query doing slow then quick, click here.
NAML

---

View this message in context: RE: Same query doing slow then quick
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.

-- 
No trees were killed in the creation of this message.
However, many electrons were terribly inconvenienced.

Thank for you answer.

shared_buffer is at 24Mb
effective_cache_size at 2048Mb

What do you mean properly ? That's not really helping a novice...




--
View this message in context:
http://postgresql.1045698.n5.nabble.com/Same-query-doing-slow-then-quick-tp5725486p5725505.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.

On 09/26/2012 16:14, FFW_Rude wrote:
> Thank for you answer.
>
> shared_buffer is at 24Mb
> effective_cache_size at 2048Mb
>
> What do you mean properly ? That's not really helping a novice...
>

from my previous mail:

before looking further, please configure shared_buffers and
effective_cache_size properly, it's fundamental
you'll probably need to raise SHMALL/SHMMAX, take a look at:
http://www.postgresql.org/docs/current/static/kernel-resources.html
for 4GB of RAM I would start with shared_buffers to 512MB and
effective_cache_size to 2GB

>
>
> --
> View this message in context:
http://postgresql.1045698.n5.nabble.com/Same-query-doing-slow-then-quick-tp5725486p5725505.html
> Sent from the PostgreSQL - performance mailing list archive at Nabble.com.
>
>


--
No trees were killed in the creation of this message.
However, many electrons were terribly inconvenienced.

Ok done to 512Mb and 2048Mb

I'm relaunching. See you in a few hours (so tommorrow)

Rude - Last Territory

Ou écouter ?

http://www.deezer.com/fr/music/last-territory/the-last-hope-3617781       (Post-apocalyptic Metal)

http://www.deezer.com/fr/music/rude-undertaker    (Pop-Rock)

Ou acheter ?

La Fnac

http://recherche.fnac.com/fmia14622213/Last-Territory

http://recherche.fnac.com/fmia14770622/Rude-Undertaker

iTunes

http://itunes.apple.com/us/artist/last-territory/id533857009?ign-mpt=uo%3D4

---

Date: Wed, 26 Sep 2012 07:17:56 -0700
From: [hidden email]
To: [hidden email]
Subject: Re: Same query doing slow then quick

On 09/26/2012 16:14, FFW_Rude wrote:
> Thank for you answer.
>
> shared_buffer is at 24Mb
> effective_cache_size at 2048Mb
>
> What do you mean properly ? That's not really helping a novice...
>

from my previous mail:

before looking further, please configure shared_buffers and
effective_cache_size properly, it's fundamental
you'll probably need to raise SHMALL/SHMMAX, take a look at:
http://www.postgresql.org/docs/current/static/kernel-resources.html
for 4GB of RAM I would start with shared_buffers to 512MB and
effective_cache_size to 2GB

>
>
> --
> View this message in context: http://postgresql.1045698.n5.nabble.com/Same-query-doing-slow-then-quick-tp5725486p5725505.html
> Sent from the PostgreSQL - performance mailing list archive at Nabble.com.
>
>


--
No trees were killed in the creation of this message.
However, many electrons were terribly inconvenienced.



--
Sent via pgsql-performance mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

jcigar.vcf (304 bytes) Download Attachment

---

If you reply to this email, your message will be added to the discussion below:

http://postgresql.1045698.n5.nabble.com/Same-query-doing-slow-then-quick-tp5725486p5725506.html

To unsubscribe from Same query doing slow then quick, click here.
NAML

---

View this message in context: RE: Same query doing slow then quick
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.

-- 
No trees were killed in the creation of this message.
However, many electrons were terribly inconvenienced.

On 26 September 2012 18:38, FFW_Rude <FFW_Rude@hotmail.com> wrote:
> root@testBI:/etc/postgresql/9.1/main# hdparm -tT /dev/sda
>  Timing cached reads:   892 MB in  2.01 seconds = 444.42 MB/sec
>  Timing buffered disk reads: 190 MB in  3.02 seconds =  62.90 MB/sec

It's OK for single HDD.


> Is fsync off by default ? I have
> #fsync = on (so it's off right ?).

Disable fsync for pgbench temporarily.


> pgbench is not found on my server. Do i have to apt-get install pgbench ?

Install the postgresql-contrib deb:
http://pkgs.org/download/postgresql-contrib

# sudo apt-get update
# sudo apt-get install postgresql-contrib


--
With best regards,
Nikolay

On 26 September 2012 19:09, FFW_Rude <FFW_Rude@hotmail.com> wrote:
> Could you explain what you are asking me to do because i don't really know
> what i'm doing...

postgresql-contrib packages contains pgbench tool on Ubuntu.

For example postgresql-contrib-9.1_9.1.3-2_i386.deb on Ubuntu 12.04 contains:
/usr/lib/postgresql/9.1/bin/pgbench


> i don't have pgbench in it. It is still pgbench command not found...

You need to run pgbench as postgres user.
For example on CentOS:
# su - postgres
$ pgbench -i pgbench && pgbench -c 10 -t 10000 pgbench


--
With best regards,
Nikolay

Sorry for the late answer, I was going through my e-mail backlog and
noticed that this question hadn't been answered.

On Thu, Sep 27, 2012 at 11:33 AM, Undertaker Rude <ffw_rude@hotmail.com> wrote:
> Oh ok. But what is this command doing ? i'm gonna runn it today. I'll keep
> you posted. Here is some EXPLAIN ANALYZE from the querys :
>
>
> Nested Loop  (cost=0.00..353722.89 rows=124893 width=16) (actual
> time=261158.061..10304193.501 rows=99 loops=1)
>   Join Filter: ((t2."X" >= (t1.x_min)::double precision) AND (t2."X" <=
> (t1.x_max)::double precision) AND (t2."Y" >= (t1.y_min)::double precision)
> AND (t2."Y" <= (t1.y_max)::double precision))
>   ->  Seq Scan on gps_22 t1  (cost=0.00..3431.80 rows=177480 width=44)
> (actual time=0.036..1399.621 rows=177480 loops=1)
>   ->  Materialize  (cost=0.00..20572.83 rows=57 width=20) (actual
> time=0.012..10.274 rows=2924 loops=177480)
>         ->  Seq Scan on adresses_22 t2  (cost=0.00..20572.55 rows=57
> width=20) (actual time=1570.240..1726.376 rows=2924 loops=1)
>               Filter: ((id_maille_200m)::text = '0'::text)
> Total runtime: 10304211.648 ms

As you can see from the explain plan, postgresql is not using any
indexes here. The reason is the type mismatch between the X and x_min
columns. Use matching types between tables to enable index use. The
same goes for the id column, if the column type is integer use a
numeric literal 0 not a text literal '0'.

Regards,
Ants Aasma
--
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt
Web: http://www.postgresql-support.de