new feature: support bounded buffered consumption of Python async file streams
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
Offer a bounded consumption path that amortizes native-to-Python async crossings without converting a sequential stream into repeated Range requests.
### Problem and Solution
`AsyncFile.read(size)` calls the core `read_buffer(size)`, which can return the current underlying network buffer rather than filling the requested size. Each returned fragment crosses the Tokio/asyncio boundary and creates a Python result. A large requested size therefore does not by itself amortize small transport fragments.
An explicit buffered reader or iterator could aggregate existing network buffers within Rust under a bounded byte budget. Reuse the current core reader rather than issuing an HTTP range for every application chunk.
Acceptance criteria:
- Preserve the existing short-read, low-latency behavior of the default API.
- Define buffering limits, EOF/error delivery, seek/ranges, and cancellation.
- Compare natural sequential reads with the same consumer, recording actual request count, return-size histogram, CPU, first-byte latency, and memory.
- Verify that fewer crossings do not come from unbounded buffering or withholding small responses indefinitely.
### Additional Context
Related to #8159, which already tracks asynchronous listing crossings. This request concerns file-stream consumption.
[AsyncFile.read](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/bindings/python/src/file.rs#L469); [Core read_buffer](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/core/core/src/types/read/futures_async_reader.rs#L182)
`read(size)` and transport `chunk` are different controls. The existing unchunked reader is the baseline; forcing a chunk changes request strategy. Fewer callbacks may primarily improve CPU efficiency even when total transfer time is unchanged.
Source references are pinned to `b6cf44f7b8a1523409e0e998e478c996ac970f03`. This request describes an optimization opportunity; it does not claim a measured end-to-end speedup.
Contributor guide
Research direction
Start with AsyncFile.read in bindings/python/src/file.rs and Core read_buffer in core/core/src/types/read/futures_async_reader.rs at the pinned revision. Define the bounded reader's buffering, EOF/error, seek/range, and cancellation behavior, then compare sequential reads using request count, return-size histogram, CPU, first-byte latency, and memory; done means fewer crossings without unbounded buffering or delayed small responses.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100