Add separate `spill_batch_size` configuration
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Currently, when spilling `RecordBatch`es to disk, datafusion serializes them using Arrow IPC in units defined by the global `batch_size` configuration (number of rows). However, it may be beneficial to decouple the spill batch size from the execution batch size. While large batches are good for vectorized execution, they can cause issues when reading back during multi-level merge (e.g. query failures even with fewer streams).
### Describe the solution you'd like
1. Add a `spill_batch_size` configuration option. (maybe in rows or maybe in bytes unit)
2. Benchmark and validate its effect on I/O throughput and query stability (failures in multi-level merge).
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.