apache / apache/opendal

new feature: add efficient io.Copy paths to Go Reader and Writer

Open
#8,248 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Implement idiomatic bulk-copy integration through `Writer.ReadFrom` and, where useful, `Reader.WriteTo`.

### Problem and Solution

The Go binding exposes Read and Write but lacks io.Copy fast-path interfaces. Each Write enters the native boundary and allocates owned storage. Ordinary file copying can consequently use 32 KiB fallback writes even though the core eventually combines them into larger storage parts. `io.CopyBuffer` alone does not guarantee a larger write: an `os.File` source implements WriterTo, which takes precedence over that supplied buffer.

A ReadFrom implementation could use a bounded reusable buffer and the existing Write operation, leaving multipart formation to the core. A corresponding WriteTo path could amortize small reads while respecting destination backpressure. Avoid recursion through io.Copy fast-path dispatch.

Acceptance criteria:
- Exercise ordinary io.Copy with a file and other readers, plus io.CopyBuffer dispatch.
- Preserve byte counts, short writes, partial reads with errors, EOF, and explicit Close.
- Keep native ownership valid after Go callers reuse input slices.
- Measure realistic file transfer, CPU, allocations, first-part latency, and memory; fewer calls do not imply proportional throughput gains.

### Additional Context

[Go Write boundary](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/bindings/go/writer.go#L646); [Go Read boundary](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/bindings/go/reader.go#L576)

This helps ordinary small-write copy paths. It does not by itself explain or fix a benchmark that already supplies explicit 8 MiB writes. Buffer size should be justified by measured tradeoffs rather than selected to favor a benchmark.

Source references are pinned to `b6cf44f7b8a1523409e0e998e478c996ac970f03`. This request describes an optimization opportunity; it does not claim a measured end-to-end speedup.

Contributor guide

Open the contributing guide

Research direction

Start with the Go binding boundaries in bindings/go/writer.go and bindings/go/reader.go at the linked Read and Write sections. Trace io.Copy and io.CopyBuffer dispatch without recursion, then add coverage for byte counts, short writes, partial reads, errors, EOF, Close, and reused input slices. Done means the fast paths preserve these behaviors and measurements cover transfer time, CPU, allocations, latency, and memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, rust
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.