apache / apache/opendal

new feature: remove the remaining AsyncWrite adapter copy from Python AsyncFile.write

Open
#8,246 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Pass immutable Python bytes to the core async Writer without first copying them through the FuturesAsyncWriter staging buffer.

### Problem and Solution

`AsyncFile.write` already retains immutable Python input with `PyBackedBytes`, but then calls `FuturesAsyncWriter.write_all(&bs)`. That adapter copies borrowed input into 256 KiB FlexBuf blocks. The core writer subsequently owns storage chunking.

The binding could retain the core `Writer` and reuse `py_bytes_like_into_buffer`, already used for owner-preserving whole-object writes, to pass an owned Buffer directly. Preserve small-write coalescing rather than mapping every small Python write to a storage request.

Acceptance criteria:
- Preserve write-all byte counts, accepted immutable input types, context-manager completion, cancellation, and close errors.
- Keep Python owners alive through buffered/in-flight/retried operations.
- Verify tiny, exact-part, non-multiple, and mixed-sized writes with full readback.
- Measure removed copy/allocation work and retained owners, then validate real file uploads without weakening SDK integrity or changing the workload.

### Additional Context

Follow-up to #8159 and merged #8171. That PR removed the initial Python-bytes-to-Vec copy and explicitly retained the 256 KiB writer adapter. This request targets the remaining adapter copy, not the already-fixed copy.

[AsyncFile state and write](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/bindings/python/src/file.rs#L430); [Existing bytes owner conversion](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/bindings/python/src/utils.rs#L29); [FuturesAsyncWriter buffering](https://github.com/apache/opendal/blob/b6cf44f7b8a1523409e0e998e478c996ac970f03/core/core/src/types/write/futures_async_writer.rs#L35)

The synchronous File flush contract should be evaluated separately; this request is scoped to AsyncFile.

Source references are pinned to `b6cf44f7b8a1523409e0e998e478c996ac970f03`. This request describes an optimization opportunity; it does not claim a measured end-to-end speedup.

Contributor guide

Open the contributing guide

Research direction

Start with AsyncFile.write in bindings/python/src/file.rs and compare its use of py_bytes_like_into_buffer in bindings/python/src/utils.rs with FuturesAsyncWriter in core/core/src/types/write/futures_async_writer.rs. Trace ownership, buffering, cancellation, close errors, and retry behavior before evaluating the scoped AsyncFile change. Done means the listed write patterns, readback, owner retention, real uploads, and copy/allocation measurements pass without weakening SDK integrity.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.