perf: push_down_filter is pathologically slow for some plans
- 主要言語
- Rust
- スター
- 9.3k
- フォーク
- 2.4k
- 平均マージ
- 3日 11時間
- マージ済み PR(30日)
- 362
説明
### Describe the bug
While investigating #17261 it became apparent that one of the largest consumers of cpu time during planning of the sql_planner_extended benchmark is the PushDownFilter OptimizerRule. I've instrumented datafusion with some logging and ran the benchmark with the following cmd:
`RUST_LOG=info cargo samply --profile=release-nonlto --bench sql_planner_extended -- --nocapture --sample-size 10`
The full output can be seen in [this gist](https://gist.github.com/Omega359/978e208b401f6af03fdf00fd8af63938) but below is the pertinent bit:
```
[2026-01-25T15:40:20Z INFO datafusion_optimizer::optimizer] Optimization (round 0) for rule push_down_limit took > 50ms: 90ms
[2026-01-25T15:43:14Z INFO datafusion_optimizer::optimizer] Optimization (round 0) for rule push_down_filter took > 50ms: 174174ms
[2026-01-25T15:43:15Z INFO datafusion_optimizer::optimizer] Optimization (round 0) for rule single_distinct_aggregation_to_group_by took > 50ms: 164ms
[2026-01-25T15:43:15Z INFO datafusion_optimizer::optimizer] Optimization (round 0) for rule eliminate_group_by_constant took > 50ms: 159ms
[2026-01-25T15:43:16Z INFO datafusion_optimizer::optimizer] Optimization (round 0) for rule common_sub_expression_eliminate took > 50ms: 1313ms
[2026-01-25T15:43:20Z INFO datafusion_optimizer::optimizer] Optimization (round 0) for rule optimize_projections took > 50ms: 3389ms
```
As you can see quite a few optimizer rules are using too much cpu for planning however the push_down_filter is the most egregious taking 174 seconds to complete. You can see from a screenshot of the output of samply where it seems most of that time is going.
### To Reproduce
`RUST_LOG=info cargo samply --profile=release-nonlto --bench sql_planner_extended -- --nocapture --sample-size 10`
Branch with profiling log @ https://github.com/Omega359/arrow-datafusion/tree/profile_optimize
### Expected behavior
Plan optimization should not be exponentially slow for some logical plans.
### Additional context
_No response_
コントリビューションガイド
調査の方向性
まず、提供された cargo samply コマンドで sql_planner_extended ベンチマークを実行し、プロファイリング出力を確認します。報告されたプランニングのホットスポットについて、PushDownFilter OptimizerRule と profile_optimize ブランチを調査します。影響を受ける論理プランで、ベンチマークに病的に長いプランニング時間が表示されなくなれば完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- rust
- 領域
- performance
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 停滞
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 35/100