Avoid masking errors and producing suboptimal plans
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
`Optimizer` returns early when a rewrite produces a certain plan for the second time.
Producing same plan again is considered a problem. Returning early is to prevent (avoidable) failure for the user, however
- it can masks actual problems (we don't know when optimization finishes early)
- it can lead to suboptimal plan being executed, which may be very expensive -- for example containing cross join because predicates weren't pushed into the join yet
### Describe the solution you'd like
Do not mask problems. Let the code fail fast, so that bugs are clear.
As interim solution, we can have a flag allowing user to recover by restoring current lenient behavior.
This can be modelled after existing `datafusion.optimizer.skip_failed_rules` config which serves similar purpose.
### Describe alternatives you've considered
_No response_
### Additional context
originates from https://github.com/apache/datafusion/pull/5623#discussion_r1664186593
cc @mslapek @alamb
Contributor guide
Research direction
Start by locating the Optimizer path that returns early when a rewrite produces the same plan twice, then compare its behavior with the existing datafusion.optimizer.skip_failed_rules configuration. Done means repeated plans no longer silently mask optimization failures by default, while a documented opt-out preserves the current lenient behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100