apache / apache/datafusion

Avoid masking errors and producing suboptimal plans

Open
#11,285 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

`Optimizer` returns early when a rewrite produces a certain plan for the second time.
Producing same plan again is considered a problem. Returning early is to prevent (avoidable) failure for the user, however

- it can masks actual problems (we don't know when optimization finishes early)
- it can lead to suboptimal plan being executed, which may be very expensive -- for example containing cross join because predicates weren't pushed into the join yet

### Describe the solution you'd like

Do not mask problems. Let the code fail fast, so that bugs are clear.
As interim solution, we can have a flag allowing user to recover by restoring current lenient behavior.

This can be modelled after existing `datafusion.optimizer.skip_failed_rules` config which serves similar purpose.

### Describe alternatives you've considered

_No response_

### Additional context

originates from https://github.com/apache/datafusion/pull/5623#discussion_r1664186593
cc @mslapek @alamb

Contributor guide

Open the contributing guide

Research direction

Start by locating the Optimizer path that returns early when a rewrite produces the same plan twice, then compare its behavior with the existing datafusion.optimizer.skip_failed_rules configuration. Done means repeated plans no longer silently mask optimization failures by default, while a documented opt-out preserves the current lenient behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.