Thread: BUG #16089: Index only scan does not happen but expected
The following bug has been logged on the website:
Bug reference: 16089
Logged by: Stepan Yankevych
Email address: stepya@ukr.net
PostgreSQL version: 11.5
Operating system: CentOS Linux release 7.3.1611 (Core)
Description:
Not a real issue but rather performance leak.
The issue is reproducible on the version 11.5 and 12.0 as well.
See script of partitioned table. Please notice PK and index contains
start_date_id as second field.
CREATE TABLE if not exists dwh.l1_snapshot_tail2 (
l1_snapshot_id int8 NOT NULL,
start_date_id int4 NOT NULL,
transaction_id int8 NOT NULL,
exchange_id varchar(6) NULL,
instrument_id int4 NULL,
bid_price numeric(12,4) NULL,
ask_price numeric(12,4) NULL,
bid_quantity int8 NULL,
ask_quantity int8 NULL,
dataset_id int4 NULL,
transaction_time timestamp NULL,
CONSTRAINT pk_l1_snapshot PRIMARY KEY (l1_snapshot_id, start_date_id)
)
PARTITION BY RANGE (start_date_id);
CREATE INDEX if not exists l1_snapshot_transact_date_idx ON
dwh.l1_snapshot_tail2 USING btree (transaction_id, start_date_id);
CREATE TABLE partitions.l1_snapshot_201805 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20180501) TO (20180601);
CREATE TABLE partitions.l1_snapshot_201806 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20180601) TO (20180701);
CREATE TABLE partitions.l1_snapshot_201807 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20180701) TO (20180801);
CREATE TABLE partitions.l1_snapshot_201808 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20180801) TO (20180901);
CREATE TABLE partitions.l1_snapshot_201809 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20180901) TO (20181001);
CREATE TABLE partitions.l1_snapshot_201810 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20181001) TO (20181101);
CREATE TABLE partitions.l1_snapshot_201811 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20181101) TO (20181201);
CREATE TABLE partitions.l1_snapshot_201812 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20181201) TO (20190101);
i have loaded 500000 rows with date_id = 20181112
Real table has 20-40 millions records a day.
Examine query
select start_date_id, count(start_date_id) as cn
from dwh.l1_snapshot_tail2
where start_date_id between 20181112 and 20181112
group by start_date_id
The execution plan shows reading full partitions.l1_snapshot_201811
Why do we need to read data from table.
We have all needed information in the index that is smaller.
I would expect index only scan (something like Oracle version of index fast
full scan )
Thank you for the report. > On Wed, Oct 30, 2019 at 12:54:31PM +0000, PG Bug reporting form wrote: > The following bug has been logged on the website: > > Bug reference: 16089 > Logged by: Stepan Yankevych > Email address: stepya@ukr.net > PostgreSQL version: 11.5 > Operating system: CentOS Linux release 7.3.1611 (Core) > Description: > > Not a real issue but rather performance leak. The issue is > reproducible on the version 11.5 and 12.0 as well. Does it mean, that on the previous versions you observed different behaviour? > The execution plan shows reading full partitions.l1_snapshot_201811 > Why do we need to read data from table. We have all needed information > in the index that is smaller. I would expect index only scan > (something like Oracle version of index fast full scan ) After a few experiments with this schema it looks like planner sometimes prefers seq scan (parallel seq scan) instead of using the index due to random read being more costly than sequential reads, even if it's necessary to read more pages. And in fact at least in my tests this was indeed faster. If you want to try out, it's possible to set random_page_cost lower and seq_page_cost higher, and planner will most likely choose a different plan, but whether it would be better or not is not clear.
>> And in fact at least in my tests this was indeed faster Can you show execution plan when index is used? Does it uses Index only scan or index scan and table scan ? STEPAN YANKEVYCH Software Engineering Team Leader Software Engineering Manager OCA Office: +380 322 424 642 x 58840 Cell: +380 96 915 9551 Email: stepan_yankevych@epam.com Lviv, Ukraine epam.com CONFIDENTIALITY CAUTION AND DISCLAIMER This message is intended only for the use of the individual(s) or entity(ies) to which it is addressed and contains informationthat is legally privileged and confidential. If you are not the intended recipient, or the person responsiblefor delivering the message to the intended recipient, you are hereby notified that any dissemination, distributionor copying of this communication is strictly prohibited. All unintended recipients are obliged to delete thismessage and destroy any printed copies. -----Original Message----- From: Dmitry Dolgov <9erthalion6@gmail.com> Sent: Wednesday, October 30, 2019 16:13 To: stepya@ukr.net; pgsql-bugs@lists.postgresql.org Subject: Re: BUG #16089: Index only scan does not happen but expected Thank you for the report. > On Wed, Oct 30, 2019 at 12:54:31PM +0000, PG Bug reporting form wrote: > The following bug has been logged on the website: > > Bug reference: 16089 > Logged by: Stepan Yankevych > Email address: stepya@ukr.net > PostgreSQL version: 11.5 > Operating system: CentOS Linux release 7.3.1611 (Core) > Description: > > Not a real issue but rather performance leak. The issue is > reproducible on the version 11.5 and 12.0 as well. Does it mean, that on the previous versions you observed different behaviour? > The execution plan shows reading full partitions.l1_snapshot_201811 > Why do we need to read data from table. We have all needed information > in the index that is smaller. I would expect index only scan > (something like Oracle version of index fast full scan ) After a few experiments with this schema it looks like planner sometimes prefers seq scan (parallel seq scan) instead ofusing the index due to random read being more costly than sequential reads, even if it's necessary to read more pages.And in fact at least in my tests this was indeed faster. If you want to try out, it's possible to set random_page_costlower and seq_page_cost higher, and planner will most likely choose a different plan, but whether it wouldbe better or not is not clear.
> On Wed, Oct 30, 2019 at 02:18:09PM +0000, Stepan Yankevych wrote: > > >> And in fact at least in my tests this was indeed faster > Can you show execution plan when index is used? > Does it uses Index only scan or index scan and table scan? Yes, at the bottom of the plan there was an Index Only Scan going through l1_snapshot_201811 (although while populating the table I've set values only for not nullable columns).
> On Wed, Oct 30, 2019 at 04:10:06PM +0100, Dmitry Dolgov wrote: > > On Wed, Oct 30, 2019 at 02:18:09PM +0000, Stepan Yankevych wrote: > > > > >> And in fact at least in my tests this was indeed faster > > Can you show execution plan when index is used? > > Does it uses Index only scan or index scan and table scan? > > Yes, at the bottom of the plan there was an Index Only Scan going > through l1_snapshot_201811 (although while populating the table I've set > values only for not nullable columns). To clarify, Index Only Scan was there only after changing random_page_cost/seq_page_cost. As I mentioned in the previous email, original plan was also parallel seq scan.
It would be great to have sequential scan of index in Postgres like we have Index Fast Full Scan in Oracle. Oracle Index FFS reads whole index and only index in the same way as full table scan (by several sequential blocks (definedby parameter) ) STEPAN YANKEVYCH Software Engineering Team Leader Software Engineering Manager OCA Office: +380 322 424 642 x 58840 Cell: +380 96 915 9551 Email: stepan_yankevych@epam.com Lviv, Ukraine epam.com CONFIDENTIALITY CAUTION AND DISCLAIMER This message is intended only for the use of the individual(s) or entity(ies) to which it is addressed and contains informationthat is legally privileged and confidential. If you are not the intended recipient, or the person responsiblefor delivering the message to the intended recipient, you are hereby notified that any dissemination, distributionor copying of this communication is strictly prohibited. All unintended recipients are obliged to delete thismessage and destroy any printed copies. -----Original Message----- From: Dmitry Dolgov <9erthalion6@gmail.com> Sent: Wednesday, October 30, 2019 17:15 To: Stepan Yankevych <Stepan_Yankevych@epam.com> Cc: stepya@ukr.net; pgsql-bugs@lists.postgresql.org Subject: Re: BUG #16089: Index only scan does not happen but expected > On Wed, Oct 30, 2019 at 04:10:06PM +0100, Dmitry Dolgov wrote: > > On Wed, Oct 30, 2019 at 02:18:09PM +0000, Stepan Yankevych wrote: > > > > >> And in fact at least in my tests this was indeed faster > > Can you show execution plan when index is used? > > Does it uses Index only scan or index scan and table scan? > > Yes, at the bottom of the plan there was an Index Only Scan going > through l1_snapshot_201811 (although while populating the table I've > set values only for not nullable columns). To clarify, Index Only Scan was there only after changing random_page_cost/seq_page_cost. As I mentioned in the previousemail, original plan was also parallel seq scan.