FileStream: Buffer more than one `FileOpenFuture`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)
Follow up to https://github.com/apache/arrow-datafusion/pull/5161#discussion_r1097183956
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
Instead of prefetching just the next stream, buffer up to `N` (where N is configurable?) open files. So maybe break this into two pipelined streams `FileOpenStream` and `FileDecodeStream` then we can used `StreamExt::buffered` to parallelize the opening.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
Do nothing
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Research direction
Start with the FileStream and FileOpenFuture entry points, then review the linked pull-request discussion for the existing prefetch behavior. Determine how FileOpenStream, FileDecodeStream, and StreamExt::buffered could fit together, and define how configurable N should work; done means multiple file opens can be buffered and parallelized without changing decoding behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100