tracking: add idiomatic bounded-memory local-file transfer APIs to bindings
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
Track the gap between generic stream primitives and a complete local-file upload/download workflow in language bindings.
### Problem and Solution
AWS CLI 2.33.15 provides a concrete application: its upload/download submitters pass a filename to the native transfer manager. The file destination allows the manager to own scheduling and writes; the application does not reconstruct an ordered array of Range responses. OpenDAL applications should have an idiomatic, bounded-memory way to complete the same file task without assembling another service-specific transfer manager.
Scope:
- Inventory the best existing public path in Python, Java, Node.js and Go before adding APIs. Existing stream wrappers and io.Copy remain valid capabilities (#8248); do not claim they are absent.
- Prioritize a native file-input path where it removes language-level copying or source preparation, and a file-output path where positioned writes can exploit independent ranges.
- Reuse core Reader/Writer, file-service and existing transfer capabilities as appropriate. Keep generic stream transformations supported.
- Specify source mutation/replay, destination overwrite/temporary-file publication, close/durability, cancellation, retry ownership, metadata and memory limits before finalizing method names.
- Let an actual caller and a measured capability gap justify each binding addition. Avoid introducing a binding-specific S3 scheduler solely to satisfy a benchmark.
Validation must compare the same final file/object result using the SDK native file API and OpenDAL best public path. Preserve integrity and include the agreed local I/O and completion semantics. Separate time to completion, CPU, memory and temporary-disk cost. No speedup is asserted by this tracking issue.
Related: #8251, #8246, #8248 and #8250.
### Additional Context
[Source reference](https://github.com/aws/aws-cli/blob/efb08f03493b8b4c71e6f66fcb9a5f89317bcb69/awscli/customizations/s3/s3handler.py). OpenDAL API inventory was checked at `b6cf44f7b8a1523409e0e998e478c996ac970f03`.
Contributor guide
Research direction
Start by inventorying the best public paths in Python, Java, Node.js and Go, using the OpenDAL API inventory at commit b6cf44f7b8a1523409e0e998e478c996ac970f03 and the AWS CLI s3handler.py reference. Compare native file APIs with the existing stream and io.Copy paths, then define the required mutation, publication, durability, cancellation, retry, metadata and memory semantics. Done means an agreed API proposal and validation plan comparing integrity, completion time, CPU, memory and temporary-disk cost.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, java, node.js, python, rust
- Domain
- api, developer-experience
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100