read: avoid repeated open and close for handle based services
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
OpenDAL's current raw read path models `Access::read(path, OpRead { range, .. })` as opening one concrete range stream.
This works naturally for HTTP object stores because each range read is a separate request. However, handle based services such as `fs`, `hdfs`, `sftp`, and `monoiofs` can reuse an opened handle and execute multiple positioned reads against it. The current core reader planner still calls `Access::read` for every planned range, so those services repeatedly open and close handles for workloads like `Reader::read`, `Reader::fetch`, chunked reads, and seek based async readers.
We should change the raw read contract so `Access::read(path, OpRead)` returns a reusable raw reader for the path and read condition set, and move byte ranges to methods on that returned reader.
Contributor guide
Research direction
Start with the raw read contract at Access::read(path, OpRead) and trace how the core reader planner uses it for each planned range. The change is complete when the returned raw reader can perform multiple byte-range reads, allowing handle-based services such as fs, hdfs, sftp, and monoiofs to reuse an opened handle.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, backend-api-design
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100