Thread: [GENERAL] Performance issue with Pointcloud extension

[GENERAL] Performance issue with Pointcloud extension

From
Eric Lemoine
Date:
Hi

We have a rather strange performance issue with the Pointcloud extension
[*]. The issue/bug may be in the extension, but we don't know for sure
at this point. I'm writing to the list to hopefully get some guidance on
how to further debug this.

[*] <https://github.com/pgpointcloud/pointcloud>

A query takes around 250 ms when executed first on a database
connection. But it takes like 3 s when executed after a first very
simple Pointcloud query.

Below is a test-case with psql.

Case #1 (works normally):

psql (9.6.3)
Type "help" for help.

lopocs=# \timing
Timing is on.
lopocs=# select points from public.sthelens where pc_intersects(points,
st_geomfromtext('polygon ((-2357334.41980829 -3742654.00016992,
-2356120.91980829 -3742654.00016992, -2356120.91980829
-3741278.00016992, -2357334.41980829 -3741278.00016992,
-2357334.41980829 -3742654.00016992))', 4978));
Time: 236.423 ms


Case #2 (works abnormally):

psql (9.6.3)
Type "help" for help.

lopocs=# \timing
Timing is on.
lopocs=# select pc_typmod_pcid(1);
 pc_typmod_pcid
----------------
              1
(1 row)

Time: 4.917 ms
lopocs=# select points from public.sthelens where pc_intersects(points,
st_geomfromtext('polygon ((-2357334.41980829 -3742654.00016992,
-2356120.91980829 -3742654.00016992, -2356120.91980829
-3741278.00016992, -2357334.41980829 -3741278.00016992,
-2357334.41980829 -3742654.00016992))', 4978));
Time: 2987.491 ms


The query takes 236 ms in case #1, and 2987 ms in case #2! Huge difference.

Anyone has any idea where this performance drop may come from? The
problem may be in the Pointcloud in the extension, but I have no idea
where the bug may be.

Any idea? Any suggestion on how to debug this? This has been driving us
crazy for some time now.

Thanks.


--
Éric Lemoine
Oslandia

Attachment

Re: [GENERAL] Performance issue with Pointcloud extension

From
Bill Moran
Date:
On Thu, 8 Jun 2017 18:00:04 +0200
Eric Lemoine <eric.lemoine@oslandia.com> wrote:

> We have a rather strange performance issue with the Pointcloud extension
> [*]. The issue/bug may be in the extension, but we don't know for sure
> at this point. I'm writing to the list to hopefully get some guidance on
> how to further debug this.
>
> [*] <https://github.com/pgpointcloud/pointcloud>
>
> A query takes around 250 ms when executed first on a database
> connection. But it takes like 3 s when executed after a first very
> simple Pointcloud query.
>
> Below is a test-case with psql.
>
> Case #1 (works normally):
>
> psql (9.6.3)
> Type "help" for help.
>
> lopocs=# \timing
> Timing is on.
> lopocs=# select points from public.sthelens where pc_intersects(points,
> st_geomfromtext('polygon ((-2357334.41980829 -3742654.00016992,
> -2356120.91980829 -3742654.00016992, -2356120.91980829
> -3741278.00016992, -2357334.41980829 -3741278.00016992,
> -2357334.41980829 -3742654.00016992))', 4978));
> Time: 236.423 ms
>
>
> Case #2 (works abnormally):
>
> psql (9.6.3)
> Type "help" for help.
>
> lopocs=# \timing
> Timing is on.
> lopocs=# select pc_typmod_pcid(1);
>  pc_typmod_pcid
> ----------------
>               1
> (1 row)
>
> Time: 4.917 ms
> lopocs=# select points from public.sthelens where pc_intersects(points,
> st_geomfromtext('polygon ((-2357334.41980829 -3742654.00016992,
> -2356120.91980829 -3742654.00016992, -2356120.91980829
> -3741278.00016992, -2357334.41980829 -3741278.00016992,
> -2357334.41980829 -3742654.00016992))', 4978));
> Time: 2987.491 ms
>
>
> The query takes 236 ms in case #1, and 2987 ms in case #2! Huge difference.
>
> Anyone has any idea where this performance drop may come from? The
> problem may be in the Pointcloud in the extension, but I have no idea
> where the bug may be.
>
> Any idea? Any suggestion on how to debug this? This has been driving us
> crazy for some time now.

Lots of missing information here ...

Is there an index on public.sthelens.points? How many rows are in that table?
What are your shared_buffers settings? How much RAM does the server have?
What does EXPLAIN look like for that query? How large (in bytes) are the
tables in question? What does pc_typmod_pcid() actually do?

There are probably lots of other questions I could ask, but those questions
are based on the fact that this _looks_ like a classic cache blowout. I.e.,
the query runs quickly when all the related data is in RAM, but is
significantly slower when the data has to be pulled from disk. Answering
the quesitons above will likely help to determine if my guess is correct.

If my guess is correct, there are any number of potential ways to improve
things: Add RAM to the machine, enlarge shared_buffers, put a geo index on
public.sthelens.points so it doesn't have to scan the entire table; as a
few examples.

Understanding what pc_typmod_pcid() actually does would help, but even
without that you can test things in a few ways. One would be to substitute
a different query in your testing for select pc_typmod_pcid(1) that is
known to push the contents of public.sthelens out of memory and see if
the behavior is similar. Any count(*) query on some other large table
would probably suffice. A better way would probalby be to install the
pg_buffercache module and see what's actually in the cache at each step
of the testing process.

In any event, if your testing doesn't help any; you'll probably need to
include answers to at least the above questions before the list will be
much help.

That is, of course, unless someone familar with pointcloud has seen this
exact problem and already knows the answer ...

--
Bill Moran <wmoran@potentialtech.com>


Re: [GENERAL] Performance issue with Pointcloud extension

From
Adrian Klaver
Date:
On 06/08/2017 09:00 AM, Eric Lemoine wrote:
> Hi
>
> We have a rather strange performance issue with the Pointcloud extension
> [*]. The issue/bug may be in the extension, but we don't know for sure
> at this point. I'm writing to the list to hopefully get some guidance on
> how to further debug this.
>
> [*] <https://github.com/pgpointcloud/pointcloud>
>
> A query takes around 250 ms when executed first on a database
> connection. But it takes like 3 s when executed after a first very
> simple Pointcloud query.
>
> Below is a test-case with psql.
>
> Case #1 (works normally):
>
> psql (9.6.3)
> Type "help" for help.
>
> lopocs=# \timing
> Timing is on.
> lopocs=# select points from public.sthelens where pc_intersects(points,
> st_geomfromtext('polygon ((-2357334.41980829 -3742654.00016992,
> -2356120.91980829 -3742654.00016992, -2356120.91980829
> -3741278.00016992, -2357334.41980829 -3741278.00016992,
> -2357334.41980829 -3742654.00016992))', 4978));
> Time: 236.423 ms
>
>
> Case #2 (works abnormally):
>
> psql (9.6.3)
> Type "help" for help.
>
> lopocs=# \timing
> Timing is on.
> lopocs=# select pc_typmod_pcid(1);
>   pc_typmod_pcid
> ----------------
>                1
> (1 row)
>
> Time: 4.917 ms
> lopocs=# select points from public.sthelens where pc_intersects(points,
> st_geomfromtext('polygon ((-2357334.41980829 -3742654.00016992,
> -2356120.91980829 -3742654.00016992, -2356120.91980829
> -3741278.00016992, -2357334.41980829 -3741278.00016992,
> -2357334.41980829 -3742654.00016992))', 4978));
> Time: 2987.491 ms
>
>
> The query takes 236 ms in case #1, and 2987 ms in case #2! Huge difference.
>
> Anyone has any idea where this performance drop may come from? The
> problem may be in the Pointcloud in the extension, but I have no idea
> where the bug may be.
>
> Any idea? Any suggestion on how to debug this? This has been driving us
> crazy for some time now.

Looks like you also have postgis and pointcloud_postgis in mix. I would
say this may get an answer sooner here:

http://lists.osgeo.org/mailman/listinfo/pgpointcloud/

>
> Thanks.
>
>


--
Adrian Klaver
adrian.klaver@aklaver.com


Re: [GENERAL] Performance issue with Pointcloud extension

From
Eric Lemoine
Date:
> Looks like you also have postgis and pointcloud_postgis in mix. I would
> say this may get an answer sooner here:
>
> http://lists.osgeo.org/mailman/listinfo/pgpointcloud/

I am actually one of the developers of the Pointcloud extension. I
haven't been able to debug this up to now.


--
Éric Lemoine
Oslandia

Attachment

Re: [GENERAL] Performance issue with Pointcloud extension

From
Adrian Klaver
Date:
On 06/08/2017 09:20 AM, Eric Lemoine wrote:
>
>> Looks like you also have postgis and pointcloud_postgis in mix. I would
>> say this may get an answer sooner here:
>>
>> http://lists.osgeo.org/mailman/listinfo/pgpointcloud/
>
> I am actually one of the developers of the Pointcloud extension. I
> haven't been able to debug this up to now.
>

Oops on my part.


--
Adrian Klaver
adrian.klaver@aklaver.com


Re: [GENERAL] Performance issue with Pointcloud extension

From
Eric Lemoine
Date:
> Lots of missing information here ...
>
> Is there an index on public.sthelens.points?

Yes, there are.

lopocs=# \d sthelens;
                           Table "public.sthelens"
 Column |    Type    |                       Modifiers
--------+------------+-------------------------------------------------------
 id     | integer    | not null default nextval('sthelens_id_seq'::regclass)
 points | pcpatch(2) |
 morton | bigint     |
Indexes:
    "sthelens_pkey" PRIMARY KEY, btree (id)
    "sthelens_pc_envelopegeometry_idx" gist (pc_envelopegeometry(points))

So two indices, one for the primary key, and a Postgis index on the
Postgis geometry returned by the Pointcloud pc_envelopegeometry function.


> How many rows are in that table?

30971



> What are your shared_buffers settings?

128 MB (Debian unstable)


> How much RAM does the server have?

16 GB

> What does EXPLAIN look like for that query? How large (in bytes) are the
> tables in question?


                                                             QUERY PLAN



----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on sthelens  (cost=383.26..2496.67 rows=2065 width=32)
(actual time=3.213..46.674 rows=2506 loops=1)
   Recheck Cond:

('01030000207213000001000000050000002E47BC352BFC41C164910500DF8D4CC12E47BC75CCF941C164910500DF8D4CC12E47BC75CCF941C1649105002F8B4CC12E47BC352BFC41C1649105002F8B4CC12E47BC352BFC41C164910500DF8D4CC1'::geometry
&& st_geomfromewkb(pc_envelopeasbinary(points)))
   Filter:

_st_intersects('01030000207213000001000000050000002E47BC352BFC41C164910500DF8D4CC12E47BC75CCF941C164910500DF8D4CC12E47BC75CCF941C1649105002F8B4CC12E47BC352BFC41C1649105002F8B4CC12E47BC352BFC41C164910500DF8D4CC1'::geometry,
st_geomfromewkb(pc_envelopeasbinary(points)))
   Heap Blocks: exact=36
   ->  Bitmap Index Scan on sthelens_pc_envelopegeometry_idx
(cost=0.00..382.75 rows=6196 width=0) (actual time=1.626..1.626
rows=2506 loops=1)
         Index Cond:

('01030000207213000001000000050000002E47BC352BFC41C164910500DF8D4CC12E47BC75CCF941C164910500DF8D4CC12E47BC75CCF941C1649105002F8B4CC12E47BC352BFC41C1649105002F8B4CC12E47BC352BFC41C164910500DF8D4CC1'::geometry
&& st_geomfromewkb(pc_envelopeasbinary(points)))
 Planning time: 0.525 ms
 Execution time: 46.999 ms
(8 rows)


Note that the execution time is 46 ms when the query is wrapped in an
explain analyze (while it's 3 s when it's not!)


> What does pc_typmod_pcid() actually do?

It is one of the simplest functions of Pointcloud.


See
<https://github.com/pgpointcloud/pointcloud/blob/master/pgsql/pointcloud.sql.in#L44-L47>
and
<https://github.com/pgpointcloud/pointcloud/blob/eb4fe1e923179ad1ca718d5620b2f41ee1a94886/pgsql/pc_pgsql.c#L132-L139>.


>
> There are probably lots of other questions I could ask, but those questions
> are based on the fact that this _looks_ like a classic cache blowout. I.e.,
> the query runs quickly when all the related d> ata is in RAM, but is
> significantly slower when the data has to be pulled from disk. Answering
> the quesitons above will likely help to determine if my guess is correct.


I don't know. The query is fast if I run it first on the database
connection. And it is *always* very slow after the "select
pc_typmod_pcid(1)" query has run.


>
> If my guess is correct, there are any number of potential ways to improve
> things: Add RAM to the machine, enlarge shared_buffers, put a geo index on
> public.sthelens.points so it doesn't have to scan the entire table; as a
> few examples.


It sounds like a performance issue to you, while it sounds like a bug to
me :)

>
> Understanding what pc_typmod_pcid() actually does would help, but even
> without that you can test things in a few ways. One would be to substitute
> a different query in your testing for select pc_typmod_pcid(1) that is
> known to push the contents of public.sthelens out of memory and see if
> the behavior is similar. Any count(*) query on some other large table
> would probably suffice. A better way would probalby be to install the
> pg_buffercache module and see what's actually in the cache at each step
> of the testing process.


I'll try to use pg_buffercache.



Thanks a lot for your response. That at least gives me courage in
debugging this :)


--
Éric Lemoine
Oslandia
+33 1 86 95 95 55

Attachment

Re: [GENERAL] Performance issue with Pointcloud extension

From
Eric Lemoine
Date:
> Note that the execution time is 46 ms when the query is wrapped in an
> explain analyze (while it's 3 s when it's not!)


Actually, it seems to me that the performance issue is not on the query
itself, it is on the fetching of the data returned by the query. Which
explains why the query is fast when executed in an explain analyze. I've
observed this by using a cursor.

The query returns 2506 rows. I use a cursor to fetch the resulting rows
500 by 500. The fetching of 500 rows (fetch 500 from c) takes about 50
ms in the good/normal case, i.e. when the "select pc_typmod_pcid(1)" is
not executed first. While it takes around 600 ms in the pathological case!

Below is the full test case.


Good case:

psql (9.6.3)
Type "help" for help.

lopocs=# \timing
Timing is on.
lopocs=# begin;
BEGIN
Time: 0.373 ms
lopocs=# declare c cursor for select points from public.sthelens where
pc_intersects(points, st_geomfromtext('polygon ((-2357334.41980829
-3742654.00016992, -2356120.91980829 -3742654.00016992,
-2356120.91980829 -3741278.00016992, -2357334.41980829
-3741278.00016992, -2357334.41980829 -3742654.00016992))', 4978));
DECLARE CURSOR
Time: 75.976 ms
lopocs=# fetch 500 from c;


            Time: 44.648 ms
lopocs=# fetch 500 from c;
Time: 40.693 ms
lopocs=# fetch 500 from c;
Time: 45.218 ms


Base case:

psql (9.6.3)
Type "help" for help.

lopocs=# \timing
Timing is on.
lopocs=# select pc_typmod_pcid(1);  -- that screws up everything
 pc_typmod_pcid
----------------
              1
(1 row)

Time: 5.702 ms
lopocs=# begin;
BEGIN
Time: 0.234 ms
lopocs=# declare c cursor for select points from public.sthelens where
pc_intersects(points, st_geomfromtext('polygon ((-2357334.41980829
-3742654.00016992, -2356120.91980829 -3742654.00016992,
-2356120.91980829 -3741278.00016992,
-2357334.41980829 -3741278.00016992, -2357334.41980829
-3742654.00016992))', 4978));
DECLARE CURSOR
Time: 76.806 ms
lopocs=# fetch 500 from c;
Time: 669.834 ms
lopocs=# fetch 500 from c;
Time: 652.738 ms
lopocs=# fetch 500 from c;
Time: 604.293 ms




How can such a thing happen? Thanks for any insight on what could cause
this.


--
Éric Lemoine
Oslandia
+33 1 86 95 95 55

Attachment

Re: [GENERAL] Performance issue with Pointcloud extension

From
Moreno Andreo
Date:
Il 08/06/2017 19:10, Eric Lemoine ha scritto:
>
> How can such a thing happen? Thanks for any insight on what could cause
> this.
>
>
I'd try raising shared_buffers to 1 GB or something near 40% of the
available memory

If you run the query again, after getting bad results, what do you get?

Cheers

Moreno.




Re: [GENERAL] Performance issue with Pointcloud extension

From
Eric Lemoine
Date:
On 06/08/2017 07:27 PM, Moreno Andreo wrote:
> Il 08/06/2017 19:10, Eric Lemoine ha scritto:
>>
>> How can such a thing happen? Thanks for any insight on what could cause
>> this.
>>
>>
> I'd try raising shared_buffers to 1 GB or something near 40% of the
> available memory

I tried to make it 4G, but it does not make a difference. My machine has
16G of RAM.


>
> If you run the query again, after getting bad results, what do you get?

Always bad results.

psql (9.6.3)
Type "help" for help.

lopocs=# \timing
Timing is on.
lopocs=# select pc_typmod_pcid(1);
 pc_typmod_pcid
----------------
              1
(1 row)

Time: 4.887 ms
lopocs=# select points from public.sthelens where pc_intersects(points,
st_geomfromtext('polygon ((-2357334.41980829 -3742654.00016992,
-2356120.91980829 -3742654.00016992, -2356120.91980829
-3741278.00016992, -2357334.41980829 -3741278.00016992,
-2357334.41980829 -3742654.00016992))', 4978));


Time: 3522.135 ms
lopocs=# select points from public.sthelens where pc_intersects(points,
st_geomfromtext('polygon ((-2357334.41980829 -3742654.00016992,
-2356120.91980829 -3742654.00016992, -2356120.91980829
-3741278.00016992, -2357334.41980829 -3741278.00016992,
-2357334.41980829 -3742654.00016992))', 4978));
Time: 3395.672 ms
lopocs=# select points from public.sthelens where pc_intersects(points,
st_geomfromtext('polygon ((-2357334.41980829 -3742654.00016992,
-2356120.91980829 -3742654.00016992, -2356120.91980829
-3741278.00016992, -2357334.41980829 -3741278.00016992,
-2357334.41980829 -3742654.00016992))', 4978));

Time: 3454.466 ms



The initial "select pc_typmod_pcid(1)" query completely screws the
connection.

"select pc_typmod_pcid(1)" is just an example of a simple query that
triggers the problem. There are many others. But it has to be a query
using the Pointcloud extension.

I have no problem if I start with the main query (my "select points
from" query of interest). And running the "select pc_typmod_pcid(1)"
query in the middle does not cause any problem. It has to be run first
on the connection to do the harm. See below.

psql (9.6.3)
Type "help" for help.

lopocs=# \timing
Timing is on.
lopocs=# select points from public.sthelens where pc_intersects(points,
st_geomfromtext('polygon ((-2357334.41980829 -3742654.00016992,
-2356120.91980829 -3742654.00016992, -2356120.91980829
-3741278.00016992, -2357334.41980829 -3741278.00016992,
-2357334.41980829 -3742654.00016992))', 4978));
Time: 280.117 ms
lopocs=# select points from public.sthelens where pc_intersects(points,
st_geomfromtext('polygon ((-2357334.41980829 -3742654.00016992,
-2356120.91980829 -3742654.00016992, -2356120.91980829
-3741278.00016992, -2357334.41980829 -3741278.00016992,
-2357334.41980829 -3742654.00016992))', 4978));
Time: 210.080 ms
lopocs=# select points from public.sthelens where pc_intersects(points,
st_geomfromtext('polygon ((-2357334.41980829 -3742654.00016992,
-2356120.91980829 -3742654.00016992, -2356120.91980829
-3741278.00016992, -2357334.41980829 -3741278.00016992,
-2357334.41980829 -3742654.00016992))', 4978));
Time: 233.095 ms
lopocs=# select pc_typmod_pcid(1);
 pc_typmod_pcid
----------------
              1
(1 row)

Time: 0.686 ms
lopocs=# select points from public.sthelens where pc_intersects(points,
st_geomfromtext('polygon ((-2357334.41980829 -3742654.00016992,
-2356120.91980829 -3742654.00016992, -2356120.91980829
-3741278.00016992, -2357334.41980829 -3741278.00016992,
-2357334.41980829 -3742654.00016992))', 4978));
Time: 199.150 ms




--
Éric Lemoine
Oslandia

Attachment

Re: [GENERAL] Performance issue with Pointcloud extension

From
Tom Lane
Date:
Eric Lemoine <eric.lemoine@oslandia.com> writes:
> The initial "select pc_typmod_pcid(1)" query completely screws the
> connection.
> "select pc_typmod_pcid(1)" is just an example of a simple query that
> triggers the problem. There are many others. But it has to be a query
> using the Pointcloud extension.

My guess is that it's got nothing to do with the specific query, but
that Pointcloud is installing some hook functions when it's loaded,
and that one or another of those hooks is killing performance for
subsequent queries, either by taking too long in itself or by
defeating some important optimization.  Hard to speculate further
with just this much data.

> I have no problem if I start with the main query (my "select points
> from" query of interest). And running the "select pc_typmod_pcid(1)"
> query in the middle does not cause any problem. It has to be run first
> on the connection to do the harm.

Have you experimented with other queries that don't involve PostGIS?
I'm wondering if your hook-installation code fails to work properly
unless PostGIS was loaded first.  This would be easier to credit if
there are hooks both extensions try to get into.

            regards, tom lane


Re: [GENERAL] Performance issue with Pointcloud extension

From
Éric
Date:


>Have you experimented with other queries that don't involve PostGIS?
>I'm wondering if your hook-installation code fails to work properly
>unless PostGIS was loaded first.  This would be easier to credit if
>there are hooks both extensions try to get into.


I think you're right on Tom. It looks like I cannot reproduce the issue if I start by calling a PostGIS function rather
thana Pointcloud function. So it may well be a conflict between PostGIS and Pointcloud. Both use fn_extra, and that
makeswe wonder. This old thread [*] makes me wonder too! I still need to figure out the bug, but I can see some light
now!thanks 

[*]
<https://www.postgresql.org/message-id/flat/984D0F47C5FF4D0DB0D71A4F6EF670ED%40cleverelephant.ca#984D0F47C5FF4D0DB0D71A4F6EF670ED@cleverelephant.ca>

--
Éric


Re: [GENERAL] Performance issue with Pointcloud extension

From
Eric Lemoine
Date:
On 06/08/2017 10:41 PM, Éric wrote:
>
>
>
>> Have you experimented with other queries that don't involve PostGIS?
>> I'm wondering if your hook-installation code fails to work properly
>> unless PostGIS was loaded first.  This would be easier to credit if
>> there are hooks both extensions try to get into.
>
>
> I think you're right on Tom. It looks like I cannot reproduce the issue if I start by calling a PostGIS function
ratherthan a Pointcloud function. So it may well be a conflict between PostGIS and Pointcloud. Both use fn_extra, and
thatmakes we wonder. This old thread [*] makes me wonder too! I still need to figure out the bug, but I can see some
lightnow! thanks 
>
> [*]
<https://www.postgresql.org/message-id/flat/984D0F47C5FF4D0DB0D71A4F6EF670ED%40cleverelephant.ca#984D0F47C5FF4D0DB0D71A4F6EF670ED@cleverelephant.ca>


I now think that the performance bug is not related to the fn_extra
thing. I had hope but not anymore :) I don't see where the Pointcloud
and PostGIS extensions could conflict.

--
Éric Lemoine
Oslandia
+33 1 86 95 95 55

Attachment

Re: [GENERAL] Performance issue with Pointcloud extension

From
Adrian Klaver
Date:
On 06/09/2017 09:13 AM, Eric Lemoine wrote:
> On 06/08/2017 10:41 PM, Éric wrote:
>>
>>
>>
>>> Have you experimented with other queries that don't involve PostGIS?
>>> I'm wondering if your hook-installation code fails to work properly
>>> unless PostGIS was loaded first.  This would be easier to credit if
>>> there are hooks both extensions try to get into.
>>
>>
>> I think you're right on Tom. It looks like I cannot reproduce the issue if I start by calling a PostGIS function
ratherthan a Pointcloud function. So it may well be a conflict between PostGIS and Pointcloud. Both use fn_extra, and
thatmakes we wonder. This old thread [*] makes me wonder too! I still need to figure out the bug, but I can see some
lightnow! thanks 
>>
>> [*]
<https://www.postgresql.org/message-id/flat/984D0F47C5FF4D0DB0D71A4F6EF670ED%40cleverelephant.ca#984D0F47C5FF4D0DB0D71A4F6EF670ED@cleverelephant.ca>
>
>
> I now think that the performance bug is not related to the fn_extra
> thing. I had hope but not anymore :) I don't see where the Pointcloud
> and PostGIS extensions could conflict.
>

Crank up the logging detail?:

https://www.postgresql.org/docs/9.6/static/runtime-config-logging.html#RUNTIME-CONFIG-LOGGING-WHEN

log_min_messages (enum)

https://www.postgresql.org/docs/9.6/static/runtime-config-logging.html#RUNTIME-CONFIG-LOGGING-WHAT

log_connections (boolean)

log_disconnections (boolean)

log_duration (boolean)


--
Adrian Klaver
adrian.klaver@aklaver.com


Re: [GENERAL] Performance issue with Pointcloud extension

From
Jeff Janes
Date:
On Fri, Jun 9, 2017 at 9:13 AM, Eric Lemoine <eric.lemoine@oslandia.com> wrote:
On 06/08/2017 10:41 PM, Éric wrote:
>
>
>
>> Have you experimented with other queries that don't involve PostGIS?
>> I'm wondering if your hook-installation code fails to work properly
>> unless PostGIS was loaded first.  This would be easier to credit if
>> there are hooks both extensions try to get into.
>
>
> I think you're right on Tom. It looks like I cannot reproduce the issue if I start by calling a PostGIS function rather than a Pointcloud function. So it may well be a conflict between PostGIS and Pointcloud. Both use fn_extra, and that makes we wonder. This old thread [*] makes me wonder too! I still need to figure out the bug, but I can see some light now! thanks
>
> [*] <https://www.postgresql.org/message-id/flat/984D0F47C5FF4D0DB0D71A4F6EF670ED%40cleverelephant.ca#984D0F47C5FF4D0DB0D71A4F6EF670ED@cleverelephant.ca>


I now think that the performance bug is not related to the fn_extra
thing. I had hope but not anymore :) I don't see where the Pointcloud
and PostGIS extensions could conflict.

Can you run 'perf top' on the slow query?  That might pretty quickly tell you which function is taking up your time.

Cheers,

Jeff

Re: [GENERAL] Performance issue with Pointcloud extension

From
Tom Lane
Date:
Jeff Janes <jeff.janes@gmail.com> writes:
> On Fri, Jun 9, 2017 at 9:13 AM, Eric Lemoine <eric.lemoine@oslandia.com>
> wrote:
>> I now think that the performance bug is not related to the fn_extra
>> thing. I had hope but not anymore :) I don't see where the Pointcloud
>> and PostGIS extensions could conflict.

> Can you run 'perf top' on the slow query?  That might pretty quickly tell
> you which function is taking up your time.

Another idea, if you haven't tried it already, is to run these test cases
in a server built with --enable-debug and --enable-cassert.  The memory
clobber stuff that's enabled by the latter is very good at turning coding
errors into reproducible, debuggable crashes ;-)

            regards, tom lane


Re: [GENERAL] Performance issue with Pointcloud extension

From
Eric Lemoine
Date:
> Another idea, if you haven't tried it already, is to run these test cases
> in a server built with --enable-debug and --enable-cassert.  The memory
> clobber stuff that's enabled by the latter is very good at turning coding
> errors into reproducible, debuggable crashes ;-)
>
>             regards, tom lane


We found the cause of the issue! Both the PostGIS and the Pointcloud
extensions define the hexbytes_from_bytes function, and the PostGIS
version is faster. The fix involves prefixing the function name in
Pointcloud, and using a similar implementation as PostGIS [*].

Thanks a lot for helping me fix that issue.

[*] <https://github.com/pgpointcloud/pointcloud/pull/185>


--
Éric Lemoine
Oslandia



Attachment