tracking: improve cross-language file and stream transfer efficiency
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
## Goal
Improve OpenDAL's latency, CPU usage, and memory footprint for idiomatic file and stream transfers across language bindings. Evaluate the same externally observable user task against the best applicable SDK API, preserving integrity, cancellation, errors, and memory limits.
## Scoped implementation work
- [ ] #8244 — optimize bulk writes in Java OperatorOutputStream
- [ ] #8245 — avoid synchronous lookahead after Java InputStream fills the destination
- [ ] #8246 — remove the remaining AsyncWrite adapter copy from Python AsyncFile.write
- [ ] #8247 — support bounded buffered consumption of Python async file streams
- [ ] #8248 — add efficient io.Copy paths to Go Reader and Writer
- [ ] #8249 — let Go Reader return available bytes without filling every caller buffer
- [ ] #8250 — reduce copies and runtime crossings in Node.js streaming reads
Related existing work: #8159 tracks Python I/O improvements; #6226 tracks Java options coverage; #7909 tracks caller-owned backend read buffers. Reuse those efforts where applicable rather than duplicating them.
## Follow-up investigations
These are investigation items, not established service defects or promised speedups. Split out implementation issues once a profile or concrete caller requirement identifies the responsible component.
- [ ] **C1 production/transfer overlap.** The [non-concurrent task path](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/core/core/src/raw/futures_util.rs#L233) awaits work inline. For a serial producer, preparing the next input can wait for the current part to complete. Measure source-read time, request intervals, and consumer work before prototyping bounded overlap. Preserve a one-request-in-flight contract when that is the scenario; distinguish it from one application transfer. Prefer the existing unchunked reader for natural sequential consumption before optimizing a forced range workflow.
- [ ] **Transfer-phase memory attribution.** Record live payload owners, allocator active/resident memory, HTTP/TLS buffers, and stage-specific RSS during concurrent transfers. Separate client initialization and benchmark-only verification. Successful task inputs are [already released](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/core/core/src/raw/futures_util.rs#L208); do not diagnose retained successful parts from process-lifetime RSS alone. Evaluate bounded buffer reuse only if allocation churn is material. [Exact writer staging](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/core/core/src/types/context/write.rs#L164) is a possible extra chunk, not evidence of an arbitrarily large memory saving; changes must preserve small-object PUT selection, retry ownership, and error timing.
- [ ] **Idiomatic local-file integration.** Start from concrete call sites needing file upload or download. Evaluate bounded native file input, Java read-option parity under #6226, and a file destination that can accept positioned writes where appropriate. Compare the SDK's native filename/WriterAt workflow, not a custom ordered Range adapter. Reuse core Reader/Writer capabilities; define source mutation, overwrite, durability, cancellation, and memory contracts before adding APIs. Keep application stream transformations supported.
## Benchmark acceptance criteria
- Identify a representative caller workflow and its source/sink before choosing a benchmark matrix. Ordered checksum consumption and download-to-file are separate scenarios.
- Include existing natural streaming APIs. An application read size is not a transport range size, and one application transfer is not the same as internal concurrency one.
- Use the actual recommended public SDK paths and resolved configuration, including integrity settings. Preserve required checksums on both sides and explicitly report capability differences.
- Separate symmetric cold startup from reused-client transfer measurements. Warm the actual measured client/transport.
- Use minimal implementation-specific workers and isolate benchmark-only readback verification, while completing that verification before accepting a result.
- Alternate paired runs without competing load; retain exact revisions, commands, bytes/digests, resource limits, and measurement boundaries.
- Validate narrow changes on the real storage workflow before expanding the matrix. Report end-to-end time, first-byte latency, CPU, and memory independently. Fewer copies or calls alone are not a throughput claim; shared network/disk/consumer limits may leave wall time unchanged.
The first implementation tranche is Java stream behavior, Python async writer ownership, and Go copy integration. Native read-boundary work follows with explicit latency and ownership contracts. Core scheduling and memory changes require attribution before implementation.
Contributor guide
Research direction
Start with scoped issues #8244–#8250 and their named entry points: Java OperatorOutputStream, Python AsyncFile.write, and Go Reader/Writer. For core investigations, read core/core/src/raw/futures_util.rs and core/core/src/raw/types/context/write.rs, then define a representative workflow and benchmark it using the stated acceptance criteria. Done requires concrete implementation issues or evidence-backed follow-ups, with integrity, cancellation, memory, and benchmark results preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, java, node.js, python, rust
- Domain
- api, backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100