apache / apache/datafusion

Consider deprecation (or removal) of `PhysicalExpr::evaluate_selection`

Open
#18,499 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Recent changes in the implementation of `CaseExpr` have replaced the usages of `PhysicalExpr::evaluate_selection` with custom filtering and selective evaluation logic. This was done to avoid overhead introduced by the generic approach `evaluate_selection` takes where it first filters the incoming record batch using the selection vector, calls plain `evaluate` with the filtered record batch, and then expands the result back to the length of the original record batch.

Because of all these changes, `PhysicalExpr::evaluate_selection` is no longer used at all in DataFusion itself. Since it's no longer used, and its use can be a cause of performance overhead, it might be better to deprecate (or even remove) it.

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

`evaluate_selection` was first introduced in #2068

As can be seen from the discussion at https://github.com/apache/arrow-rs/issues/3620 actually implementing selective evaluation more efficiently is not trivial. Does it make sense to have this as an overridable trait function?

Contributor guide

Open the contributing guide

Research direction

Start by locating the PhysicalExpr::evaluate_selection trait method and checking whether any DataFusion implementations or call sites remain; then inspect the CaseExpr selective-evaluation changes described in the issue. Done means reaching and documenting a clear decision on deprecation versus removal, including any affected API users or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.