Move the spill pre-processing steps into `SpillManager`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
I noticed it when reviewing https://github.com/apache/datafusion/pull/18207
When writing spill files in the sort executor, it requires some pre-processing steps for the batches to spill to avoid duplicate writes, specifically https://github.com/apache/datafusion/blob/167baf718278112d427910a0296f366909bc3384/datafusion/physical-plan/src/sorts/sort.rs#L409
This step should apply to all spilling usages, not only limited to the sort executor, so a better way is to put it inside `SpillManager`/`InProgressSpillFile`, or maybe `Arrow` IPC writer if possible.
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start with the spill pre-processing step at datafusion/physical-plan/src/sorts/sort.rs around line 409, then trace SpillManager and InProgressSpillFile to understand their current responsibilities. Consider whether the common behavior belongs there or in the Arrow IPC writer. Done means spill preprocessing is shared by all spilling usages rather than only the sort executor, while avoiding duplicate writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100