apache / apache/datafusion

Support FilterExec with embedded projections in WindowTopN optimization

Open
#21,596 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

[Discussion thread](https://github.com/apache/datafusion/pull/21479#discussion_r3068891661)

The `WindowTopN` physical optimizer rule currently skips `FilterExec` nodes that have an embedded projection (`filter.projection().is_some()`). This happens when `ProjectionPushdown` runs before `WindowTopN` and embeds a projection into the `FilterExec`.

```rust
// Don't handle filters with projections
if filter.projection().is_some() {
return None;
}
```

### Describe the solution you'd like

Extend WindowTopN to handle FilterExec with embedded projections. This would require:

1. Extracting the projection mapping from FilterExec
2. Resolving the window column index through the projection (mapping projected column indices back to the original schema)
3. Carrying the projection forward into the replacement plan (either re-wrapping PartitionedTopKExec with a ProjectionExec, or embedding the projection)

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at the WindowTopN physical optimizer rule and inspect how FilterExec exposes an embedded projection and how the window column is resolved. Reproduce the ProjectionPushdown-before-WindowTopN case, then verify the replacement plan preserves the projection while applying the WindowTopN optimization.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.