apache / apache/datafusion

Support spilling for WindowAggExec

Open
#22,946 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

`WindowAggExec` can require buffering an entire window partition before it can evaluate some window functions. Today that buffering is memory-only, so queries over large or skewed partitions can exhaust the configured memory pool even when the runtime has a spill-capable disk manager.

This is surprising because other memory-intensive physical operators in DataFusion already participate in spill paths, but window aggregation does not. A query with a large `PARTITION BY` group, or a query without `PARTITION BY`, can therefore fail with `Resources exhausted` rather than using the configured spill storage.

### Motivation sequence

The desired behavior is for `WindowAggExec` to keep the existing partition-at-a-time evaluation model, but use disk spill as a fallback when a complete window partition no longer fits in the memory reservation.

```mermaid
sequenceDiagram
autonumber
participant Input as Sorted input stream
participant Window as WindowAggExec
participant Memory as MemoryReservation
participant Spill as SpillManager / DiskManager
participant Expr as Window expression evaluator
participant Output as Downstream consumer

Input->>Window: RecordBatch with ordered window partitions
loop For each logical window partition
Window->>Memory: Reserve memory for buffered partition rows
alt Reservation succeeds
Window->>Window: Buffer partition rows in memory
else Reservation fails and disk spill is enabled
Window->>Spill: Write buffered rows to spill file
Window->>Memory: Release memory for spilled rows
Input->>Window: Continue same partition
Window->>Spill: Append remaining partition rows
else Reservation fails and spill is unavailable
Window-->>Output: Return ResourcesExhausted
end

Window->>Spill: Read spilled rows, if partition spilled
Window->>Expr: Evaluate window functions for complete partition
Expr-->>Window: Window result columns
Window-->>Output: Emit original columns plus window columns
end
```

### Describe the solution you'd like

Add spill support to `WindowAggExec`:

- Track buffered partition batches with a `MemoryReservation`.
- Preserve the current sorted-input partition semantics and finish one partition at a time.
- When the active partition cannot grow its reservation and a disk manager is available, write the buffered partition batches to spill files through the existing `SpillManager`.
- Read spilled partitions back when the partition is ready for window expression evaluation.
- Report spill metrics such as spill count, spilled rows, and spilled bytes.
- Keep the non-spill path unchanged when memory is sufficient.

### Describe alternatives you've considered

A more advanced alternative would be implementing streaming evaluation for more window frame/function combinations. That can reduce memory further for specific functions, but it is a larger semantic change and does not cover all window functions. Operator-level spill support is still useful as a general fallback for large partitions.

### Additional context

This would make `WindowAggExec` behave more consistently with DataFusion's other spill-aware physical operators and make memory-limited window queries fail less often when spill storage is configured.

Contributor guide

Open the contributing guide

Research direction

Start by reading WindowAggExec and the existing spill-aware physical operators, then trace how MemoryReservation, SpillManager, and DiskManager are used. The work is complete when WindowAggExec can spill and reload oversized partitions, preserve partition-at-a-time evaluation, report spill metrics, and retain the current in-memory path when capacity is sufficient.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.