new feature: reduce adapter overhead in Python synchronous File writes
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
Reduce the synchronous Python File writer's adapter buffering and repeated runtime entry while preserving the existing file-like API.
### Problem and proposed direction
At `4b132b0d33c8a5b84ff83f6fcfbab264860c5b3d`, the path is `File.write -> StdWriter.write_all -> StdWriter.write -> Handle.block_on(FuturesAsyncWriter.write)`. The FuturesAsyncWriter stages borrowed input through a 256 KiB FlexBuf, even when Python supplies an immutable 8 MiB bytes object. A large File.write can therefore require repeated adapter writes and runtime entries before the storage writer handles multipart chunking.
The binding could retain the core blocking Writer and pass an owned Buffer retaining the immutable Python bytes. `py_bytes_like_into_buffer` already provides an owner-preserving conversion for whole-object writes. This is a proposal to remove intermediate work, not a claim of measured benefit from an unimplemented change.
[Python File](https://github.com/apache/opendal/blob/4b132b0d33c8a5b84ff83f6fcfbab264860c5b3d/bindings/python/src/file.rs), [StdWriter](https://github.com/apache/opendal/blob/4b132b0d33c8a5b84ff83f6fcfbab264860c5b3d/core/core/src/blocking/write/std_writer.rs), [FuturesAsyncWriter](https://github.com/apache/opendal/blob/4b132b0d33c8a5b84ff83f6fcfbab264860c5b3d/core/core/src/types/write/futures_async_writer.rs).
### Observed workload
Measured OpenDAL commit: `4b132b0d33c8a5b84ff83f6fcfbab264860c5b3d`; harness commit: `c4c203859ed72d9a0ffac7438486a2d205649102`. Real S3 in us-east-2, c7i.8xlarge with a 4-vCPU/1-GiB client limit, six formal repeats. Boto3 1.43.88 uses native APIs, without a transfer manager. Values below are medians, not significance claims. The harness and raw archive are currently private, so these numbers are descriptive evidence, not a publicly reproducible benchmark package.
Uploading a 59,158,238-byte local file uses 8 MiB producer blocks, 8 MiB multipart parts and CRC32C on both sides, followed by independently verified readback. At C4, median elapsed time was 0.419 s for OpenDAL File and 0.370 s for Boto3, while process CPU was lower for OpenDAL (0.075 s versus 0.124 s). These measurements motivate investigation but do not isolate the adapter as the cause of the elapsed-time gap.
### Acceptance criteria
- Preserve accepted inputs, write-all byte counts, small-write coalescing, flush behavior, close errors and context-manager completion across services.
- Retain immutable Python owners through buffered, in-flight and retried writes. Do not weaken isolation for mutable inputs.
- Validate tiny, exact-part, non-multiple and mixed-size writes with full readback.
- Measure copy/allocation work, runtime entries and retained memory, then compare real-S3 file uploads under unchanged conditions; request timing should separate dispatch and completion delays from copying.
Related: #8246 intentionally covers AsyncFile and leaves the synchronous flush contract separate. #8159 covers releasing the GIL during blocking File I/O; that work is complementary and should remain separately attributable.
Contributor guide
Research direction
Read bindings/python/src/file.rs, core/core/src/blocking/write/std_writer.rs, and core/core/src/types/write/futures_async_writer.rs, starting with the synchronous File.write path and py_bytes_like_into_buffer. Define and measure the buffering, runtime-entry, allocation, and ownership behavior before comparing unchanged-condition real-S3 uploads; done requires the listed write, flush, close, ownership, isolation, and readback criteria across services.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100