tracking: improve Python binding I/O performance
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
## Summary
Several Python binding paths perform avoidable full-payload copies, add an extra storage request, or cross the Tokio-to-asyncio boundary once per result. Blocking APIs also keep the CPython GIL while waiting for storage I/O. These costs affect large-object throughput, async listing, remote-read latency, and Python thread concurrency.
Source references are against [`7ca2c0261`](https://github.com/apache/opendal/commit/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41).
## Work items
### Fix sized `readline`
- [ ] Fix `File.readline(size)` and add regression coverage. The current implementation starts with `size` zero bytes, appends the line, and then truncates using the number of appended bytes. For example, reading `b"abc\ndef"` with `readline(4)` returns `b"\x00\x00\x00\x00"`. ([implementation](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/file.rs#L158-L174), [existing coverage](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/tests/test_read.py#L94-L118))
### Reduce payload copies
- [ ] Return Python `bytes` from `Operator.read` and `AsyncOperator.read` with one full-payload copy. The current path copies `opendal::Buffer` into `Vec`, creates a temporary Python buffer object, and copies it again into Python `bytes`. ([sync](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/operator.rs#L472-L478), [async](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/operator.rs#L1213-L1224), [conversion](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/utils.rs#L24-L52))
- [ ] Avoid allocating and zero-initializing the complete requested size before `File.read(size)`. Avoid repeated vector growth for `read()` when the length is already known. Preserve partial-read and EOF behavior. ([sync](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/file.rs#L99-L117), [async](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/file.rs#L493-L515))
- [ ] Let immutable Python `bytes` enter `Operator.write` and `AsyncOperator.write` without a binding-owned payload copy. Keep the Python object alive until the storage operation completes; mutable inputs must remain isolated. ([sync](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/operator.rs#L527-L561), [async](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/operator.rs#L1278-L1316))
- [ ] Remove the initial Python-bytes-to-`Vec` copy in `AsyncFile.write`. Preserve the existing small-write coalescing behavior so the change does not increase storage requests or file syscalls. ([binding](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/file.rs#L530-L558), [writer buffer](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/core/core/src/types/write/futures_async_writer.rs#L35-L47))
### Reduce async listing overhead
- [ ] Avoid one Tokio-to-asyncio round trip per `AsyncLister` entry. Fetch or prefetch entries in batches and serve buffered entries without another cross-thread event-loop wakeup. Preserve streaming and bounded-memory behavior. ([implementation](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/lister.rs#L55-L87))
### Avoid an extra request when opening readers
- [ ] Do not require `stat`/`HEAD` before a normal sequential `open(path, "rb")` read. The current file adapter resolves an unbounded range by fetching the object length before reading. Preserve bounded-range reads and seek behavior. ([binding](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/operator.rs#L1073-L1105), [range resolution](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/core/core/src/types/context/read.rs#L116-L164))
### Release the GIL during blocking I/O
- [ ] Release the CPython GIL while blocking Operator methods wait for storage I/O. Ensure borrowed Python inputs remain alive and immutable for the complete operation. Extend the same behavior to File and blocking-list operations when their state can be transferred safely.
`#[pymodule(gil_used = false)]` declares support for free-threaded Python, but it does not release the GIL for blocking calls on regular CPython. The binding currently has no `Python::detach` or `allow_threads` path.
## Evidence
A release build on CPython 3.14.5 with the GIL enabled used the memory backend to isolate binding overhead. These results indicate where copies and runtime crossings dominate; they do not predict remote-storage throughput.
- Reading an 8 MiB object took 195.4 us with `Operator.read` and 94.5 us with `File.readinto` using a preallocated buffer.
- Reading the same 8 MiB object took 233.7 us with `AsyncOperator.read` and 1,047.3 us through `AsyncFile.read`.
- Listing 2,000 in-memory entries took 0.613 us per entry synchronously and 40.14 us per entry asynchronously.
- The async list produced 2,002 event-loop wakeups: one for lister creation, one per entry, and one for completion.
## Validation
- [ ] Add focused benchmarks that keep Operator and event-loop setup outside the measured operation. The current benchmark includes both in every timed run and combines multiple object sizes into one result. ([current benchmark](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/benchmark/async_opendal_benchmark.py#L45-L80))
- [ ] Measure payload copies and allocation volume for changed read and write paths.
- [ ] Measure runtime crossings and event-loop wakeups for async listing.
- [ ] Verify request count against a real HTTP object store for the reader-open change.
- [ ] Verify blocking calls allow another Python thread to make progress.
- [ ] Run affected tests and benchmarks on regular CPython, the Python 3.11 abi3 wheel, and free-threaded Python 3.14.
## Completion criteria
- Whole-object reads create Python `bytes` with one full-payload copy.
- Immutable Python `bytes` do not require a binding-owned payload copy before whole-object writes.
- Async listing materially reduces runtime crossings and event-loop wakeups per entry without collecting an unbounded result set.
- Sequential file reads do not add `stat`/`HEAD` unless the requested operation needs the object length.
- Blocking storage I/O does not hold the GIL.
- File cursor, partial-read, buffering, close/durability, cancellation, and free-threaded safety behavior remain unchanged.
Contributor guide
Research direction
Start with the specifically linked Python binding files, such as bindings/python/src/file.rs, operator.rs, lister.rs, and utils.rs, and their referenced tests and benchmark. Pick one work item, reproduce its current behavior with the named coverage, then trace the related core read or writer code before changing it. Done means the selected regression or performance criterion passes without breaking the listed buffering, request, cursor, cancellation, or threading behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100