BUG #16089: Index only scan does not happen but expected

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

BUG #16089: Index only scan does not happen but expected

apt.postgresql.org Repository Update
The following bug has been logged on the website:

Bug reference:      16089
Logged by:          Stepan Yankevych
Email address:      [hidden email]
PostgreSQL version: 11.5
Operating system:   CentOS Linux release 7.3.1611 (Core)
Description:        

Not a real issue but rather performance leak.
The issue is reproducible on the version 11.5 and 12.0 as well.
See script of partitioned table. Please notice PK and index contains
start_date_id as second field.
 
CREATE TABLE if not exists dwh.l1_snapshot_tail2 (
        l1_snapshot_id int8 NOT NULL,
        start_date_id int4 NOT NULL,
        transaction_id int8 NOT NULL,
        exchange_id varchar(6) NULL,
        instrument_id int4 NULL,
        bid_price numeric(12,4) NULL,
        ask_price numeric(12,4) NULL,
        bid_quantity int8 NULL,
        ask_quantity int8 NULL,
        dataset_id int4 NULL,
        transaction_time timestamp NULL,
        CONSTRAINT pk_l1_snapshot PRIMARY KEY (l1_snapshot_id, start_date_id)
)
PARTITION BY RANGE (start_date_id);
CREATE INDEX if not exists l1_snapshot_transact_date_idx ON
dwh.l1_snapshot_tail2 USING btree (transaction_id, start_date_id);


CREATE TABLE partitions.l1_snapshot_201805 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20180501) TO (20180601);
CREATE TABLE partitions.l1_snapshot_201806 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20180601) TO (20180701);
CREATE TABLE partitions.l1_snapshot_201807 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20180701) TO (20180801);
CREATE TABLE partitions.l1_snapshot_201808 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20180801) TO (20180901);
CREATE TABLE partitions.l1_snapshot_201809 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20180901) TO (20181001);
CREATE TABLE partitions.l1_snapshot_201810 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20181001) TO (20181101);
CREATE TABLE partitions.l1_snapshot_201811 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20181101) TO (20181201);
CREATE TABLE partitions.l1_snapshot_201812 PARTITION OF
dwh.l1_snapshot_tail2 FOR VALUES FROM (20181201) TO (20190101);

i have loaded 500000 rows with date_id = 20181112
Real table has 20-40 millions records a day.

Examine query
select start_date_id, count(start_date_id) as cn
from dwh.l1_snapshot_tail2
where start_date_id between 20181112 and 20181112
group by start_date_id

The execution plan shows reading full partitions.l1_snapshot_201811
Why do we need to read data from table.
We have all needed information in the index that is smaller.
I would expect index only scan (something like Oracle version of index fast
full scan )

Reply | Threaded
Open this post in threaded view
|

Re: BUG #16089: Index only scan does not happen but expected

Dmitry Dolgov
Thank you for the report.

> On Wed, Oct 30, 2019 at 12:54:31PM +0000, PG Bug reporting form wrote:
> The following bug has been logged on the website:
>
> Bug reference:      16089
> Logged by:          Stepan Yankevych
> Email address:      [hidden email]
> PostgreSQL version: 11.5
> Operating system:   CentOS Linux release 7.3.1611 (Core)
> Description:
>
> Not a real issue but rather performance leak. The issue is
> reproducible on the version 11.5 and 12.0 as well.

Does it mean, that on the previous versions you observed different
behaviour?

> The execution plan shows reading full partitions.l1_snapshot_201811
> Why do we need to read data from table. We have all needed information
> in the index that is smaller. I would expect index only scan
> (something like Oracle version of index fast full scan )

After a few experiments with this schema it looks like planner sometimes
prefers seq scan (parallel seq scan) instead of using the index due to
random read being more costly than sequential reads, even if it's
necessary to read more pages. And in fact at least in my tests this was
indeed faster. If you want to try out, it's possible to set
random_page_cost lower and seq_page_cost higher, and planner will most
likely choose a different plan, but whether it would be better or not
is not clear.


Reply | Threaded
Open this post in threaded view
|

RE: BUG #16089: Index only scan does not happen but expected

Stepan Yankevych
>> And in fact at least in my tests this was indeed faster
Can you show execution plan when index is used?
Does it uses Index only scan or index scan and table scan ?

STEPAN YANKEVYCH
Software Engineering Team Leader
Software Engineering Manager
OCA
 
Office: +380 322 424 642 x 58840   Cell: +380 96 915 9551   Email: [hidden email]
Lviv, Ukraine   epam.com
 
 
CONFIDENTIALITY CAUTION AND DISCLAIMER
This message is intended only for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.
 


-----Original Message-----
From: Dmitry Dolgov <[hidden email]>
Sent: Wednesday, October 30, 2019 16:13
To: [hidden email]; [hidden email]
Subject: Re: BUG #16089: Index only scan does not happen but expected

Thank you for the report.

> On Wed, Oct 30, 2019 at 12:54:31PM +0000, PG Bug reporting form wrote:
> The following bug has been logged on the website:
>
> Bug reference:      16089
> Logged by:          Stepan Yankevych
> Email address:      [hidden email]
> PostgreSQL version: 11.5
> Operating system:   CentOS Linux release 7.3.1611 (Core)
> Description:
>
> Not a real issue but rather performance leak. The issue is
> reproducible on the version 11.5 and 12.0 as well.

Does it mean, that on the previous versions you observed different behaviour?

> The execution plan shows reading full partitions.l1_snapshot_201811
> Why do we need to read data from table. We have all needed information
> in the index that is smaller. I would expect index only scan
> (something like Oracle version of index fast full scan )

After a few experiments with this schema it looks like planner sometimes prefers seq scan (parallel seq scan) instead of using the index due to random read being more costly than sequential reads, even if it's necessary to read more pages. And in fact at least in my tests this was indeed faster. If you want to try out, it's possible to set random_page_cost lower and seq_page_cost higher, and planner will most likely choose a different plan, but whether it would be better or not is not clear.


Reply | Threaded
Open this post in threaded view
|

Re: BUG #16089: Index only scan does not happen but expected

Dmitry Dolgov
> On Wed, Oct 30, 2019 at 02:18:09PM +0000, Stepan Yankevych wrote:
>
> >> And in fact at least in my tests this was indeed faster
> Can you show execution plan when index is used?
> Does it uses Index only scan or index scan and table scan?

Yes, at the bottom of the plan there was an Index Only Scan going
through l1_snapshot_201811 (although while populating the table I've set
values only for not nullable columns).


Reply | Threaded
Open this post in threaded view
|

Re: BUG #16089: Index only scan does not happen but expected

Dmitry Dolgov
> On Wed, Oct 30, 2019 at 04:10:06PM +0100, Dmitry Dolgov wrote:
> > On Wed, Oct 30, 2019 at 02:18:09PM +0000, Stepan Yankevych wrote:
> >
> > >> And in fact at least in my tests this was indeed faster
> > Can you show execution plan when index is used?
> > Does it uses Index only scan or index scan and table scan?
>
> Yes, at the bottom of the plan there was an Index Only Scan going
> through l1_snapshot_201811 (although while populating the table I've set
> values only for not nullable columns).

To clarify, Index Only Scan was there only after changing
random_page_cost/seq_page_cost. As I mentioned in the previous email,
original plan was also parallel seq scan.


Reply | Threaded
Open this post in threaded view
|

RE: BUG #16089: Index only scan does not happen but expected

Stepan Yankevych

 It would be great to have sequential scan of index in Postgres like we have Index Fast Full Scan in Oracle.
 Oracle Index FFS reads whole index and only index in the same way as full table scan (by several sequential blocks (defined by parameter) )


STEPAN YANKEVYCH
Software Engineering Team Leader
Software Engineering Manager
OCA
 
Office: +380 322 424 642 x 58840   Cell: +380 96 915 9551   Email: [hidden email]
Lviv, Ukraine   epam.com
 
 
CONFIDENTIALITY CAUTION AND DISCLAIMER
This message is intended only for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.
 


-----Original Message-----
From: Dmitry Dolgov <[hidden email]>
Sent: Wednesday, October 30, 2019 17:15
To: Stepan Yankevych <[hidden email]>
Cc: [hidden email]; [hidden email]
Subject: Re: BUG #16089: Index only scan does not happen but expected

> On Wed, Oct 30, 2019 at 04:10:06PM +0100, Dmitry Dolgov wrote:
> > On Wed, Oct 30, 2019 at 02:18:09PM +0000, Stepan Yankevych wrote:
> >
> > >> And in fact at least in my tests this was indeed faster
> > Can you show execution plan when index is used?
> > Does it uses Index only scan or index scan and table scan?
>
> Yes, at the bottom of the plan there was an Index Only Scan going
> through l1_snapshot_201811 (although while populating the table I've
> set values only for not nullable columns).

To clarify, Index Only Scan was there only after changing random_page_cost/seq_page_cost. As I mentioned in the previous email, original plan was also parallel seq scan.