apache / apache/datafusion

TPC-DS 54 with dynamic filter slow due to inefficient plan

Open
#18,290 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

I'm using datafusion `50.1.0` with dynamic filter pushdown enabled. The following query is a small section of TPC-DS q54. I noticed a large slow down, and noticed in explain analyze that `web_sales` was scanned completely. The plans and the explain analyze of the scan are [here](https://gist.github.com/Mark1626/c7e54ce3c3dd5d77dac825ee3f072ab7)

```sql
select
i_item_id item_id,
sum(ws_ext_sales_price) ws_item_rev
from
web_sales,
item,
date_dim
where
ws_item_sk = i_item_sk
and d_date in (
select
d_date
from
date_dim
where
d_week_seq =(
select
d_week_seq
from
date_dim
where
d_date = '1998-02-19'
)
)
and ws_sold_date_sk = d_date_sk
group by
i_item_id
```

Dynamic filter

```
predicate=DynamicFilterPhysicalExpr [ ws_item_sk@0 >= 1 AND ws_item_sk@0 <= 54000 ] AND DynamicFilterPhysicalExpr [ ws_sold_date_sk@2 >= 2415022 AND ws_sold_date_sk@2 <= 2488070 ],
```

Explain Analyze
```
metrics=[output_rows=3598645, elapsed_compute=14ns, batches_split=0, bytes_scanned=0, file_open_errors=0, file_scan_errors=0, files_ranges_pruned_statistics=0, num_predicate_creation_errors=0, page_index_rows_matched=0, page_index_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=7197290, pushdown_rows_pruned=858, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=1800, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, bloom_filter_eval_time=41.140467ms, metadata_load_time=356.618768ms, page_index_eval_time=205.72µs, row_pushdown_eval_time=284.704072ms, statistics_eval_time=40.716318ms, time_elapsed_opening=52.71329ms, time_elapsed_processing=2.034425262s, time_elapsed_scanning_total=9.825828938s, time_elapsed_scanning_until_data=6.676775892s]
```

Same query simplified is much faster

```
select
i_item_id item_id,
sum(ws_ext_sales_price) ws_item_rev
from
web_sales,
item,
date_dim
where
ws_item_sk = i_item_sk
and d_date in (
'1998-02-17', '1998-02-18', '1998-02-19', '1998-02-20', '1998-02-21', '1998-02-22', '1998-02-23'
)
and ws_sold_date_sk = d_date_sk
group by
i_item_id;
```

Dynamic filter

A much smaller range is used now

```
predicate=DynamicFilterPhysicalExpr [ ws_item_sk@0 >= 1 AND ws_item_sk@0 <= 54000 ] AND DynamicFilterPhysicalExpr [ ws_sold_date_sk@2 >= 2450862 AND ws_sold_date_sk@2 <= 2450868 ],
```

Explain Analyze

Shows a lot less output rows as more rows pruned

```
metrics=[output_rows=7880, elapsed_compute=14ns, batches_split=0, bytes_scanned=0, file_open_errors=0, file_scan_errors=0, files_ranges_pruned_statistics=1829, num_predicate_creation_errors=0, page_index_rows_matched=0, page_index_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=15760, pushdown_rows_pruned=858, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=2, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, bloom_filter_eval_time=44.38µs, metadata_load_time=931.791µs, page_index_eval_time=4.667µs, row_pushdown_eval_time=868.499µs, statistics_eval_time=59.672µs, time_elapsed_opening=4.838903ms, time_elapsed_processing=292.590292ms, time_elapsed_scanning_total=220.44782ms, time_elapsed_scanning_until_data=216.799489ms]
```

I strongly suspect the plan and the join order as in the simplified version a lot more files ranges are pruned

Contributor guide

Open the contributing guide

Research direction

Start with the linked plans and explain-analyze output, comparing the dynamic-filter predicate ranges, scan pruning metrics, and join order between the two queries. Investigate why the TPC-DS q54 variant scans web_sales broadly; done means it achieves effective range pruning comparable to the simplified query without regressing the reported behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.