Support FilterExec with embedded projections in WindowTopN optimization
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
[Discussion thread](https://github.com/apache/datafusion/pull/21479#discussion_r3068891661)
The `WindowTopN` physical optimizer rule currently skips `FilterExec` nodes that have an embedded projection (`filter.projection().is_some()`). This happens when `ProjectionPushdown` runs before `WindowTopN` and embeds a projection into the `FilterExec`.
```rust
// Don't handle filters with projections
if filter.projection().is_some() {
return None;
}
```
### Describe the solution you'd like
Extend WindowTopN to handle FilterExec with embedded projections. This would require:
1. Extracting the projection mapping from FilterExec
2. Resolving the window column index through the projection (mapping projected column indices back to the original schema)
3. Carrying the projection forward into the replacement plan (either re-wrapping PartitionedTopKExec with a ProjectionExec, or embedding the projection)
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start at the WindowTopN physical optimizer rule and inspect how FilterExec exposes an embedded projection and how the window column is resolved. Reproduce the ProjectionPushdown-before-WindowTopN case, then verify the replacement plan preserves the projection while applying the WindowTopN optimization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100