apache / apache/datafusion

Move the spill pre-processing steps into `SpillManager`

Open
#18,261 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

I noticed it when reviewing https://github.com/apache/datafusion/pull/18207

When writing spill files in the sort executor, it requires some pre-processing steps for the batches to spill to avoid duplicate writes, specifically https://github.com/apache/datafusion/blob/167baf718278112d427910a0296f366909bc3384/datafusion/physical-plan/src/sorts/sort.rs#L409

This step should apply to all spilling usages, not only limited to the sort executor, so a better way is to put it inside `SpillManager`/`InProgressSpillFile`, or maybe `Arrow` IPC writer if possible.

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the spill pre-processing step at datafusion/physical-plan/src/sorts/sort.rs around line 409, then trace SpillManager and InProgressSpillFile to understand their current responsibilities. Consider whether the common behavior belongs there or in the Arrow IPC writer. Done means spill preprocessing is shared by all spilling usages rather than only the sort executor, while avoiding duplicate writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.