apache / apache/datafusion

Change datafusion-cli file access to use mmap

Open
#21,159 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

While looking at traces for the morsel driven scan
- #20529

@Dandandan and I noticed there is substantial potential work done on other blocking threads just to read data

I was thinking that for local files, we might do substantially better using mmap / the kernel page cache

### Describe the solution you'd like

Specifically, a new wrapper for ObjectStore that will memory map files when they are opened, and then provide a way to read (using zero copy `Bytes::slice`) from those memory mapped files.

I am not sure it would actually be faster -- so the first thing to do would be to code it up and try it out / see how fast we can get it

### Describe alternatives you've considered

Use the mmap2 crate as shown in this example: https://github.com/apache/arrow-rs/blob/main/arrow/examples/
zero_copy_ipc.rs

Open the file, meory map it, and then turn it into `Bytes` that is then zero copied when requested. At first keep
all mmap files. We will eventually implement a cap / LRU for the number of open mmaps, but to start we can just
keep them all open and see how that goes.

https://github.com/apache/arrow-rs/blob/main/arrow/examples/zero_copy_ipc.rs#L46-L47

Then we will wire this into datafusio-cli here so it is used for file urls

https://github.com/alamb/datafusion/blob/7ef62b988d19c75e737b57f1491cfc1cd9222466/datafusion-cli/src/
object_storage.rs#L567-L566

I think the mmap file object store should not wrap another object store, but instead it could implement all functions directly. Use the LocalFileSystem as an example for how to implement the various methods if needed

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in datafusion-cli/src/object_storage.rs at the file-URL integration point, then read the arrow-rs zero_copy_ipc.rs example and LocalFileSystem implementation for ObjectStore methods. Implement and evaluate a mmap-backed local-file store whose reads use zero-copy Bytes::slice, and confirm that datafusion-cli uses it for file URLs.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.