Thread: Duplicate WHERE condition changes performance and plan

On Thu, 16 Apr 2020 at 07:56, singh400@gmail.com <singh400@gmail.com> wrote:
> We have an odd issue where specifying the same where clause twice causes PG to pick a much more efficent plan. We
wouldlike to know why.
 

> The EXPLAIN ANALYZE for both queries can be found here:-
> Query A: https://explain.depesz.com/s/lFuy
> Query B: https://explain.depesz.com/s/Jqmv

This is basically down to just a poor join selectivity estimation.
The selectivity estimation on the duplicate not equal clause is not
removed by the planner and the selectivity of that is taking into
account twice which reduces the selectivity of the table named
"problem". With that selectivity taken into account, the query planner
thinks a nested loop will be a more optimal plan, to which it seems to
be.

Join selectivity estimations can use the most common values lists as
you may see if you look at the pg_stats view for the tables and
columns involved in the join condition.  Perhaps ID columns are not
good candidates to get an MCV list in the stats.  In that case, the
ndistinct estimate will be used.  If there's no MCV list in the stats
then check ndistinct is reasonably accurate. If there is an MCV list,
then you can make that bigger by increasing the statistics targets on
the join columns and running ANALYZE. Note: Planning can become slower
when you increase the statistics targets.

Starting with PostgreSQL 9.6, foreign keys are also used to help with
join selectivity estimations. I see you have a suitable foreign key
from the schema you posted.  You might want to add that to the list of
reasons to upgrade.

David

> Starting with PostgreSQL 9.6, foreign keys are also used to help with
> join selectivity estimations. I see you have a suitable foreign key
> from the schema you posted.  You might want to add that to the list of
> reasons to upgrade.

Apologies for the delay in response. I've had "PostgreSQL 9.6.3,
compiled by Visual C++ build 1800, 64-bit" setup at home for a while
and after importing the data across I'm still seeing the same
behaviour.

Even after upgrading my local install of PG to "PostgreSQL 12.2,
compiled by Visual C++ build 1914, 64-bit" and I'm still seeing the
same behaviour.

Plans for PG12:-
Query A: https://explain.depesz.com/s/zrVD
Query B: https://explain.depesz.com/s/ZLWe

The settings for my home setup are left at default, nothing special.

Indy

On Wed, Apr 15, 2020 at 08:55:53PM +0100, singh400@gmail.com wrote:
> We have an odd issue where specifying the same where clause twice causes PG
> to pick a much more efficent plan. We would like to know why.

> Query B (this is the 'fast' query):
> UPDATE problem_instance SET processed = false
> FROM problem
> WHERE problem.id = problem_instance.problem_id
> AND problem.status != 2
> AND problem.status != 2
> AND processed = true;

When you specify redundant condition, it results in an underestimate, as
expected:

Index Scan using problem_id_idx1 on public.problem (cost=0.28..624.68 ROWS=73 width=14) (actual time=0.011..0.714
ROWS=841loops=1)
 
    Filter: ((problem.status <> 2) AND (problem.status <> 2))

In this case, doing an index scans on problem_instance is apparently faster
than an seq scan.

I think the duplicate condition is fooling the planner, and by chance it's
giving a better plan.  That might indicate that your settings aren't ideal.
Maybe random_page_cost should be lower, which would encourage index scans.  If
you're using SSD storage, or if the DB is small compared with shared_buffers or
RAM, then random_page_cost should be closer to seq_page_cost.

How large are the indexes? problem_id_idx1 ?

On Mon, Apr 20, 2020 at 01:50:17AM +0100, singh400@gmail.com wrote:
> Even after upgrading my local install of PG to "PostgreSQL 12.2,
> compiled by Visual C++ build 1914, 64-bit" and I'm still seeing the
> same behaviour.

> Server Configuration:
> https://gist.github.com/indy-singh/8386d59206af042d365e5cd49fbae68f
> shared_buffers     2GB     configuration file
> effective_cache_size     6GB     configuration file

Note, until v10, the documentation said this:

https://www.postgresql.org/docs/9.6/runtime-config-resource.html
|Also, on Windows, large values for shared_buffers aren't as effective. You may
|find better results keeping the setting relatively low and using the operating
|system cache more instead. The useful range for shared_buffers on Windows
|systems is generally from 64MB to 512MB.

It would be interesting to know

-- 
Justin

> If you're using SSD storage, or if the DB is small compared with shared_buffers or RAM, then random_page_cost should
becloser to seq_page_cost.
 

I don't *think* we are using SSDs  but I'll need to confirm that though.

> How large are the indexes? problem_id_idx1 ?

Using the query from here:
https://wiki.postgresql.org/wiki/Index_Maintenance#Index_size.2Fusage_statistics
Output here: https://gist.github.com/indy-singh/e33eabe5cc937043c93b42a8783b3bfb

I've setup a repo here where it is possible to reproduce the weird
behaviour I'm getting:-

https://github.com/indy-singh/postgres-duplicate-where-conditon

That contains the data (amended to remove any private information) as
well as the statements need to recreate tables, indices, and
constraints,

I think after some trial and error this is something to do with the
size of the table and statistics. I've been trying to put together a
Short, Self Contained, Correct example (http://sscce.org/) and the
problem only appears when fill problem_instance.message with junk, but
I have to do it in two steps as outlined in the README in repo.

Indy

> I don't *think* we are using SSDs  but I'll need to confirm that though.

Confirmed we are not using SSDs but '10K RPM SAS in RAID-10.'

I've also been hunt for other queries that show this behaviour too,
and I've found one. The PG settings/versions will be different in this
example due to the earlier example being for our internal CI/CD tool
which is hosted a on local instance of PG. This example is directly
from our production servers.

Query C (slow):-
SELECT COUNT(1)
FROM proposal.proposal
INNER JOIN proposal.note ON proposal.note.proposal_reference =
proposal.proposal.reference
WHERE 1 = 1
AND proposal.proposal.system_id = 11
AND proposal.proposal.legacy_organisation_id IN (6, 7, 11, 16, 18, 44,
200, 218, 233, 237, 259, 47)
AND proposal.proposal.has_been_anonymised = false
AND proposal.note.legacy_read_by IS NULL
AND proposal.note.type_id IN (1, 4, 9)
AND proposal.note.entry_time > '2020-04-01'
AND proposal.note.entry_time < '2020-05-01';

Query D (fast):-
SELECT COUNT(1)
FROM proposal.proposal
INNER JOIN proposal.note ON proposal.note.proposal_reference =
proposal.proposal.reference
WHERE 1 = 1
AND proposal.proposal.system_id = 11
AND proposal.proposal.legacy_organisation_id IN (6, 7, 11, 16, 18, 44,
200, 218, 233, 237, 259, 47)
AND proposal.proposal.has_been_anonymised = false
AND proposal.proposal.has_been_anonymised = false
AND proposal.note.legacy_read_by IS NULL
AND proposal.note.type_id IN (1, 4, 9)
AND proposal.note.entry_time > '2020-04-01'
AND proposal.note.entry_time < '2020-05-01';

The EXPLAIN ANALYZE for both queries can be found here:-
Query C: https://explain.depesz.com/s/5Mbu
Query D: https://explain.depesz.com/s/jVnH

The table definitions (including the indexes) can be found here:-
proposal.proposal:
https://gist.github.com/indy-singh/6ccd86ff859e7cdad2ec1bf73a61445c
proposal.note: https://gist.github.com/indy-singh/6c1f85ad15cb92e138447a91d8cf3ecb

Data stats:-
proposal.proposal has 10,324,779 rows and once the table specific
conditions are applied there are 39,223 rows left.
proposal.note has 28,97,698 rows and once the table specific
conditions are applied there are 54,359 rows left.

PG version:
PostgreSQL 9.5.17 on x86_64-pc-linux-gnu (Debian 9.5.17-1.pgdg90+1),
compiled by gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516, 64-bit

-- SELECT relname, relpages, reltuples, relallvisible, relkind,
relnatts, relhassubclass, reloptions, pg_table_size(oid) FROM pg_class
WHERE relname='TABLE_NAME'
Table metadata:-
proposal.proposal:
https://gist.github.com/indy-singh/24e7ec8f3d4e2c3ac73f724cea52f9de
proposal.note: https://gist.github.com/indy-singh/104d6ec7ef8179461eb4f91c121615e0

Index Stats:-
proposal.proposal:
https://gist.github.com/indy-singh/1d41d15addb543bcdafc8641b9d7f036
proposal.note: https://gist.github.com/indy-singh/7a698dec98dd8ef2808345d1802e6b6a

Last Vacuum:-
proposal.proposal: Never
proposal.note: 2020-04-17 15:10:57.256013+01

Last Analyze:
proposal.proposal: Never
proposal.note: 2020-04-07 11:48:49.689622+01

Server Configuration:
https://gist.github.com/indy-singh/b19134873f266ee6ce2b9815504d130c

Indy