new feature: add efficient io.Copy paths to Go Reader and Writer
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
Implement idiomatic bulk-copy integration through `Writer.ReadFrom` and, where useful, `Reader.WriteTo`.
### Problem and Solution
The Go binding exposes Read and Write but lacks io.Copy fast-path interfaces. Each Write enters the native boundary and allocates owned storage. Ordinary file copying can consequently use 32 KiB fallback writes even though the core eventually combines them into larger storage parts. `io.CopyBuffer` alone does not guarantee a larger write: an `os.File` source implements WriterTo, which takes precedence over that supplied buffer.
A ReadFrom implementation could use a bounded reusable buffer and the existing Write operation, leaving multipart formation to the core. A corresponding WriteTo path could amortize small reads while respecting destination backpressure. Avoid recursion through io.Copy fast-path dispatch.
Acceptance criteria:
- Exercise ordinary io.Copy with a file and other readers, plus io.CopyBuffer dispatch.
- Preserve byte counts, short writes, partial reads with errors, EOF, and explicit Close.
- Keep native ownership valid after Go callers reuse input slices.
- Measure realistic file transfer, CPU, allocations, first-part latency, and memory; fewer calls do not imply proportional throughput gains.
### Additional Context
[Go Write boundary](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/bindings/go/writer.go#L646); [Go Read boundary](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/bindings/go/reader.go#L576)
This helps ordinary small-write copy paths. It does not by itself explain or fix a benchmark that already supplies explicit 8 MiB writes. Buffer size should be justified by measured tradeoffs rather than selected to favor a benchmark.
Source references are pinned to `b6cf44f7b8a1523409e0e998e478c996ac970f03`. This request describes an optimization opportunity; it does not claim a measured end-to-end speedup.
Contributor guide
Research direction
Start with the Go binding boundaries in bindings/go/writer.go and bindings/go/reader.go at the linked Read and Write sections. Trace io.Copy and io.CopyBuffer dispatch without recursion, then add coverage for byte counts, short writes, partial reads, errors, EOF, Close, and reused input slices. Done means the fast paths preserve these behaviors and measurements cover transfer time, CPU, allocations, latency, and memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, rust
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100