apache / apache/datafusion

Improve Spill Performance: `mmap` the spill files

Open
#15,321 10 comments 2 reactions 1 assignee Claimed by @zebsme View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

- part of https://github.com/apache/datafusion/issues/15271
### Is your feature request related to a problem or challenge?

Today when DataFusion spills files to disk, it uses the Arrow IPC format

Here is the code:
https://github.com/apache/datafusion/blob/988a53540b67cb36f3f259b47a68fe11736fccbb/datafusion/physical-plan/src/spill.rs#L60-L88

The IPC reader currently reads the spill files using file IO and into memory.

it is possible to use `mmap` to zero copy the contents of the files into memory. Here is an example of how to do so:

https://github.com/apache/arrow-rs/blob/main/arrow/examples/zero_copy_ipc.rs

- My testing on https://github.com/apache/arrow-rs/pull/7120 suggested mmap is 3x faster than file IO

### Describe the solution you'd like

I would like to see if using mmap to read the spill files back in is faster

### Describe alternatives you've considered

1. Use mmap to read spill files
3. Add / use a benchmark showing the peformance benefit of doing this

### Additional context

- https://github.com/apache/datafusion/issues/15320

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.