Change datafusion-cli file access to use mmap
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
While looking at traces for the morsel driven scan
- #20529
@Dandandan and I noticed there is substantial potential work done on other blocking threads just to read data
I was thinking that for local files, we might do substantially better using mmap / the kernel page cache
### Describe the solution you'd like
Specifically, a new wrapper for ObjectStore that will memory map files when they are opened, and then provide a way to read (using zero copy `Bytes::slice`) from those memory mapped files.
I am not sure it would actually be faster -- so the first thing to do would be to code it up and try it out / see how fast we can get it
### Describe alternatives you've considered
Use the mmap2 crate as shown in this example: https://github.com/apache/arrow-rs/blob/main/arrow/examples/
zero_copy_ipc.rs
Open the file, meory map it, and then turn it into `Bytes` that is then zero copied when requested. At first keep
all mmap files. We will eventually implement a cap / LRU for the number of open mmaps, but to start we can just
keep them all open and see how that goes.
https://github.com/apache/arrow-rs/blob/main/arrow/examples/zero_copy_ipc.rs#L46-L47
Then we will wire this into datafusio-cli here so it is used for file urls
https://github.com/alamb/datafusion/blob/7ef62b988d19c75e737b57f1491cfc1cd9222466/datafusion-cli/src/
object_storage.rs#L567-L566
I think the mmap file object store should not wrap another object store, but instead it could implement all functions directly. Use the LocalFileSystem as an example for how to implement the various methods if needed
### Additional context
_No response_
Contributor guide
Research direction
Start in datafusion-cli/src/object_storage.rs at the file-URL integration point, then read the arrow-rs zero_copy_ipc.rs example and LocalFileSystem implementation for ObjectStore methods. Implement and evaluate a mmap-backed local-file store whose reads use zero-copy Bytes::slice, and confirm that datafusion-cli uses it for file URLs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100