apache / apache/opendal

new feature: reduce adapter overhead in Python synchronous File writes

Open
#8,266 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Reduce the synchronous Python File writer's adapter buffering and repeated runtime entry while preserving the existing file-like API.

### Problem and proposed direction

At `4b132b0d33c8a5b84ff83f6fcfbab264860c5b3d`, the path is `File.write -> StdWriter.write_all -> StdWriter.write -> Handle.block_on(FuturesAsyncWriter.write)`. The FuturesAsyncWriter stages borrowed input through a 256 KiB FlexBuf, even when Python supplies an immutable 8 MiB bytes object. A large File.write can therefore require repeated adapter writes and runtime entries before the storage writer handles multipart chunking.

The binding could retain the core blocking Writer and pass an owned Buffer retaining the immutable Python bytes. `py_bytes_like_into_buffer` already provides an owner-preserving conversion for whole-object writes. This is a proposal to remove intermediate work, not a claim of measured benefit from an unimplemented change.

[Python File](https://github.com/apache/opendal/blob/4b132b0d33c8a5b84ff83f6fcfbab264860c5b3d/bindings/python/src/file.rs), [StdWriter](https://github.com/apache/opendal/blob/4b132b0d33c8a5b84ff83f6fcfbab264860c5b3d/core/core/src/blocking/write/std_writer.rs), [FuturesAsyncWriter](https://github.com/apache/opendal/blob/4b132b0d33c8a5b84ff83f6fcfbab264860c5b3d/core/core/src/types/write/futures_async_writer.rs).

### Observed workload

Measured OpenDAL commit: `4b132b0d33c8a5b84ff83f6fcfbab264860c5b3d`; harness commit: `c4c203859ed72d9a0ffac7438486a2d205649102`. Real S3 in us-east-2, c7i.8xlarge with a 4-vCPU/1-GiB client limit, six formal repeats. Boto3 1.43.88 uses native APIs, without a transfer manager. Values below are medians, not significance claims. The harness and raw archive are currently private, so these numbers are descriptive evidence, not a publicly reproducible benchmark package.

Uploading a 59,158,238-byte local file uses 8 MiB producer blocks, 8 MiB multipart parts and CRC32C on both sides, followed by independently verified readback. At C4, median elapsed time was 0.419 s for OpenDAL File and 0.370 s for Boto3, while process CPU was lower for OpenDAL (0.075 s versus 0.124 s). These measurements motivate investigation but do not isolate the adapter as the cause of the elapsed-time gap.

### Acceptance criteria

- Preserve accepted inputs, write-all byte counts, small-write coalescing, flush behavior, close errors and context-manager completion across services.
- Retain immutable Python owners through buffered, in-flight and retried writes. Do not weaken isolation for mutable inputs.
- Validate tiny, exact-part, non-multiple and mixed-size writes with full readback.
- Measure copy/allocation work, runtime entries and retained memory, then compare real-S3 file uploads under unchanged conditions; request timing should separate dispatch and completion delays from copying.

Related: #8246 intentionally covers AsyncFile and leaves the synchronous flush contract separate. #8159 covers releasing the GIL during blocking File I/O; that work is complementary and should remain separately attributable.

Contributor guide

Open the contributing guide

Research direction

Read bindings/python/src/file.rs, core/core/src/blocking/write/std_writer.rs, and core/core/src/types/write/futures_async_writer.rs, starting with the synchronous File.write path and py_bytes_like_into_buffer. Define and measure the buffering, runtime-entry, allocation, and ownership behavior before comparing unchanged-condition real-S3 uploads; done requires the listed write, flush, close, ownership, isolation, and readback criteria across services.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.