apache / apache/datafusion

Reuse file descriptors for LocalFileSystem object storage

Open
#19,983 6 comments 2 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

The ObjectStore implementation is stateless and doesn't allow for passing `File` descriptors.
This means for every request it's doing a `File::open` (and close) call, which creates overhead (mainly when doing a lot of small requests, e.g. with predicate pushdown enabled).

### Describe the solution you'd like

Implement an object store implementation for DataFusion that opens / reuses the `File`s for each path.

We can use `Arc` stored in hashmap for each file / path + read_at (stateless file read) to minimize overhead and maximize concurrency.

### Describe alternatives you've considered

_No response_

### Additional context

Also see

https://github.com/apache/arrow-rs-object-store/issues/622
https://github.com/apache/arrow-rs-object-store/issues/614

Contributor guide

Open the contributing guide

Research direction

Start with DataFusion's ObjectStore and LocalFileSystem entry points, then review the linked arrow-rs-object-store issues #622 and #614 for related constraints. The work is complete when LocalFileSystem reuses open File descriptors per path while preserving concurrent stateless reads and avoiding an open/close cycle for every request.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.