apache / apache/datafusion

Avoid rerunning logical optimizer rules on unchanged plans

Open
#22,411 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the problem

The logical optimizer reruns every rule on each optimizer pass, even when a rule already returned `Transformed::no` for the current logical plan and no later rule has changed that plan before the next invocation of the same rule.

This adds avoidable planning cost on queries that need multiple optimizer passes. ClickBench queries are a good example: many converge only after a second pass, but most rules in that pass see the exact same input they already reported as unchanged.

### Proposed improvement

Track a cheap logical plan version during optimization. Increment the version whenever a rule reports `transformed = true`. If a rule previously returned `Transformed::no` at the current plan version, skip rerunning it until some rule changes the plan.

This keeps the existing optimizer fixed-point behavior while avoiding repeated no-op work for unchanged rule inputs.

### Local measurement

Using a temporary local ClickBench logical planning probe over all 51 ClickBench queries, 30 iterations:

- Baseline optimizer-only: 607.259 us/query
- Patched optimizer-only: 409.473 us/query and 423.034 us/query
- Baseline parse/analyze/optimize: 1001.498 us/query
- Patched parse/analyze/optimize: 822.877 us/query and 844.032 us/query

The patch preserves optimizer behavior and only skips rules after a prior `Transformed::no` for the same unchanged plan version.

Contributor guide

Open the contributing guide

Research direction

Start at the logical optimizer's rule-pass entry point and trace how each rule reports Transformed::no or transformed = true. Add plan-version tracking so unchanged no-op rules are skipped until a rule changes the plan, then verify fixed-point behavior and compare the ClickBench logical-planning measurements.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.