apache / apache/opendal

read: avoid repeated open and close for handle based services

Open
#7,659 0 comments 4 reactions 0 assignees View on GitHub
enhancement services/fs services/hdfs services/sftp
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

OpenDAL's current raw read path models `Access::read(path, OpRead { range, .. })` as opening one concrete range stream.

This works naturally for HTTP object stores because each range read is a separate request. However, handle based services such as `fs`, `hdfs`, `sftp`, and `monoiofs` can reuse an opened handle and execute multiple positioned reads against it. The current core reader planner still calls `Access::read` for every planned range, so those services repeatedly open and close handles for workloads like `Reader::read`, `Reader::fetch`, chunked reads, and seek based async readers.

We should change the raw read contract so `Access::read(path, OpRead)` returns a reusable raw reader for the path and read condition set, and move byte ranges to methods on that returned reader.

Contributor guide

Open the contributing guide

Research direction

Start with the raw read contract at Access::read(path, OpRead) and trace how the core reader planner uses it for each planned range. The change is complete when the returned raw reader can perform multiple byte-range reads, allowing handle-based services such as fs, hdfs, sftp, and monoiofs to reuse an opened handle.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, backend-api-design
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.