apache / apache/opendal

new feature: support bounded buffered consumption of Python async file streams

Open
#8,247 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Offer a bounded consumption path that amortizes native-to-Python async crossings without converting a sequential stream into repeated Range requests.

### Problem and Solution

`AsyncFile.read(size)` calls the core `read_buffer(size)`, which can return the current underlying network buffer rather than filling the requested size. Each returned fragment crosses the Tokio/asyncio boundary and creates a Python result. A large requested size therefore does not by itself amortize small transport fragments.

An explicit buffered reader or iterator could aggregate existing network buffers within Rust under a bounded byte budget. Reuse the current core reader rather than issuing an HTTP range for every application chunk.

Acceptance criteria:
- Preserve the existing short-read, low-latency behavior of the default API.
- Define buffering limits, EOF/error delivery, seek/ranges, and cancellation.
- Compare natural sequential reads with the same consumer, recording actual request count, return-size histogram, CPU, first-byte latency, and memory.
- Verify that fewer crossings do not come from unbounded buffering or withholding small responses indefinitely.

### Additional Context

Related to #8159, which already tracks asynchronous listing crossings. This request concerns file-stream consumption.

[AsyncFile.read](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/bindings/python/src/file.rs#L469); [Core read_buffer](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/core/core/src/types/read/futures_async_reader.rs#L182)

`read(size)` and transport `chunk` are different controls. The existing unchunked reader is the baseline; forcing a chunk changes request strategy. Fewer callbacks may primarily improve CPU efficiency even when total transfer time is unchanged.

Source references are pinned to `b6cf44f7b8a1523409e0e998e478c996ac970f03`. This request describes an optimization opportunity; it does not claim a measured end-to-end speedup.

Contributor guide

Open the contributing guide

Research direction

Start with AsyncFile.read in bindings/python/src/file.rs and Core read_buffer in core/core/src/types/read/futures_async_reader.rs at the pinned revision. Define the bounded reader's buffering, EOF/error, seek/range, and cancellation behavior, then compare sequential reads using request count, return-size histogram, CPU, first-byte latency, and memory; done means fewer crossings without unbounded buffering or delayed small responses.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.