apache / apache/opendal

new feature: let Go Reader return available bytes without filling every caller buffer

Open
#8,249 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Improve progressive-stream latency by separating normal Read behavior from explicitly buffered bulk consumption.

### Problem and Solution

The Go Reader loops over native reads until the caller buffer is full, EOF occurs, or an error arrives. The underlying Rust reader can already return available bytes, but this loop may withhold them while waiting for more data. Large destination buffers can therefore delay application processing.

Consider returning available bytes after a native read, while using an explicit bulk-copy path for throughput-oriented aggregation. Simply removing the loop may increase Go-level calls, so evaluate latency and CPU together.

Acceptance criteria:
- A controlled reader with an available prefix and blocked subsequent data returns the prefix promptly.
- Preserve documented read semantics, partial-read/error ordering, EOF, Seek, and Close.
- Cover small and large buffers and slow consumers.
- Evaluate alongside Reader.WriteTo so improved first-byte latency does not unnecessarily inflate calls for bulk transfers.

### Additional Context

[Loop that fills the destination](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/bindings/go/reader.go#L576); [Underlying asynchronous read](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/core/core/src/types/read/futures_async_reader.rs#L265)

This is a generic binding API improvement request. A service-specific defect or an end-to-end S3 speedup is not asserted.

Source references are pinned to `b6cf44f7b8a1523409e0e998e478c996ac970f03`. This request describes an optimization opportunity; it does not claim a measured end-to-end speedup.

Contributor guide

Open the contributing guide

Research direction

Start in bindings/go/reader.go at the linked loop, then compare the underlying asynchronous read in core/core/src/types/read/futures_async_reader.rs. Add controlled-reader coverage for available-prefix, blocked-data, buffer-size, slow-consumer, error, EOF, Seek, and Close cases, and compare normal Read with Reader.WriteTo for latency and call/CPU tradeoffs.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, rust
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.