Improve Spill Performance: `mmap` the spill files
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
- part of https://github.com/apache/datafusion/issues/15271
### Is your feature request related to a problem or challenge?
Today when DataFusion spills files to disk, it uses the Arrow IPC format
Here is the code:
https://github.com/apache/datafusion/blob/988a53540b67cb36f3f259b47a68fe11736fccbb/datafusion/physical-plan/src/spill.rs#L60-L88
The IPC reader currently reads the spill files using file IO and into memory.
it is possible to use `mmap` to zero copy the contents of the files into memory. Here is an example of how to do so:
https://github.com/apache/arrow-rs/blob/main/arrow/examples/zero_copy_ipc.rs
- My testing on https://github.com/apache/arrow-rs/pull/7120 suggested mmap is 3x faster than file IO
### Describe the solution you'd like
I would like to see if using mmap to read the spill files back in is faster
### Describe alternatives you've considered
1. Use mmap to read spill files
3. Add / use a benchmark showing the peformance benefit of doing this
### Additional context
- https://github.com/apache/datafusion/issues/15320
Contributor guide
Assessment
This issue has not been assessed yet.