new feature: let Go Reader return available bytes without filling every caller buffer
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
Improve progressive-stream latency by separating normal Read behavior from explicitly buffered bulk consumption.
### Problem and Solution
The Go Reader loops over native reads until the caller buffer is full, EOF occurs, or an error arrives. The underlying Rust reader can already return available bytes, but this loop may withhold them while waiting for more data. Large destination buffers can therefore delay application processing.
Consider returning available bytes after a native read, while using an explicit bulk-copy path for throughput-oriented aggregation. Simply removing the loop may increase Go-level calls, so evaluate latency and CPU together.
Acceptance criteria:
- A controlled reader with an available prefix and blocked subsequent data returns the prefix promptly.
- Preserve documented read semantics, partial-read/error ordering, EOF, Seek, and Close.
- Cover small and large buffers and slow consumers.
- Evaluate alongside Reader.WriteTo so improved first-byte latency does not unnecessarily inflate calls for bulk transfers.
### Additional Context
[Loop that fills the destination](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/bindings/go/reader.go#L576); [Underlying asynchronous read](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/core/core/src/types/read/futures_async_reader.rs#L265)
This is a generic binding API improvement request. A service-specific defect or an end-to-end S3 speedup is not asserted.
Source references are pinned to `b6cf44f7b8a1523409e0e998e478c996ac970f03`. This request describes an optimization opportunity; it does not claim a measured end-to-end speedup.
Contributor guide
Research direction
Start in bindings/go/reader.go at the linked loop, then compare the underlying asynchronous read in core/core/src/types/read/futures_async_reader.rs. Add controlled-reader coverage for available-prefix, blocked-data, buffer-size, slow-consumer, error, EOF, Seek, and Close cases, and compare normal Read with Reader.WriteTo for latency and call/CPU tradeoffs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, rust
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100