apache / apache/opendal

tracking: improve Python binding I/O performance

Open
#8,159 1 comment 3 reactions 0 assignees View on GitHub
bindings/python enhancement
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

## Summary

Several Python binding paths perform avoidable full-payload copies, add an extra storage request, or cross the Tokio-to-asyncio boundary once per result. Blocking APIs also keep the CPython GIL while waiting for storage I/O. These costs affect large-object throughput, async listing, remote-read latency, and Python thread concurrency.

Source references are against [`7ca2c0261`](https://github.com/apache/opendal/commit/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41).

## Work items

### Fix sized `readline`

- [ ] Fix `File.readline(size)` and add regression coverage. The current implementation starts with `size` zero bytes, appends the line, and then truncates using the number of appended bytes. For example, reading `b"abc\ndef"` with `readline(4)` returns `b"\x00\x00\x00\x00"`. ([implementation](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/file.rs#L158-L174), [existing coverage](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/tests/test_read.py#L94-L118))

### Reduce payload copies

- [ ] Return Python `bytes` from `Operator.read` and `AsyncOperator.read` with one full-payload copy. The current path copies `opendal::Buffer` into `Vec`, creates a temporary Python buffer object, and copies it again into Python `bytes`. ([sync](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/operator.rs#L472-L478), [async](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/operator.rs#L1213-L1224), [conversion](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/utils.rs#L24-L52))
- [ ] Avoid allocating and zero-initializing the complete requested size before `File.read(size)`. Avoid repeated vector growth for `read()` when the length is already known. Preserve partial-read and EOF behavior. ([sync](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/file.rs#L99-L117), [async](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/file.rs#L493-L515))
- [ ] Let immutable Python `bytes` enter `Operator.write` and `AsyncOperator.write` without a binding-owned payload copy. Keep the Python object alive until the storage operation completes; mutable inputs must remain isolated. ([sync](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/operator.rs#L527-L561), [async](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/operator.rs#L1278-L1316))
- [ ] Remove the initial Python-bytes-to-`Vec` copy in `AsyncFile.write`. Preserve the existing small-write coalescing behavior so the change does not increase storage requests or file syscalls. ([binding](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/file.rs#L530-L558), [writer buffer](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/core/core/src/types/write/futures_async_writer.rs#L35-L47))

### Reduce async listing overhead

- [ ] Avoid one Tokio-to-asyncio round trip per `AsyncLister` entry. Fetch or prefetch entries in batches and serve buffered entries without another cross-thread event-loop wakeup. Preserve streaming and bounded-memory behavior. ([implementation](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/lister.rs#L55-L87))

### Avoid an extra request when opening readers

- [ ] Do not require `stat`/`HEAD` before a normal sequential `open(path, "rb")` read. The current file adapter resolves an unbounded range by fetching the object length before reading. Preserve bounded-range reads and seek behavior. ([binding](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/src/operator.rs#L1073-L1105), [range resolution](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/core/core/src/types/context/read.rs#L116-L164))

### Release the GIL during blocking I/O

- [ ] Release the CPython GIL while blocking Operator methods wait for storage I/O. Ensure borrowed Python inputs remain alive and immutable for the complete operation. Extend the same behavior to File and blocking-list operations when their state can be transferred safely.

`#[pymodule(gil_used = false)]` declares support for free-threaded Python, but it does not release the GIL for blocking calls on regular CPython. The binding currently has no `Python::detach` or `allow_threads` path.

## Evidence

A release build on CPython 3.14.5 with the GIL enabled used the memory backend to isolate binding overhead. These results indicate where copies and runtime crossings dominate; they do not predict remote-storage throughput.

- Reading an 8 MiB object took 195.4 us with `Operator.read` and 94.5 us with `File.readinto` using a preallocated buffer.
- Reading the same 8 MiB object took 233.7 us with `AsyncOperator.read` and 1,047.3 us through `AsyncFile.read`.
- Listing 2,000 in-memory entries took 0.613 us per entry synchronously and 40.14 us per entry asynchronously.
- The async list produced 2,002 event-loop wakeups: one for lister creation, one per entry, and one for completion.

## Validation

- [ ] Add focused benchmarks that keep Operator and event-loop setup outside the measured operation. The current benchmark includes both in every timed run and combines multiple object sizes into one result. ([current benchmark](https://github.com/apache/opendal/blob/7ca2c02614f3fc2f16ee976c9f18be250e4f8e41/bindings/python/benchmark/async_opendal_benchmark.py#L45-L80))
- [ ] Measure payload copies and allocation volume for changed read and write paths.
- [ ] Measure runtime crossings and event-loop wakeups for async listing.
- [ ] Verify request count against a real HTTP object store for the reader-open change.
- [ ] Verify blocking calls allow another Python thread to make progress.
- [ ] Run affected tests and benchmarks on regular CPython, the Python 3.11 abi3 wheel, and free-threaded Python 3.14.

## Completion criteria

- Whole-object reads create Python `bytes` with one full-payload copy.
- Immutable Python `bytes` do not require a binding-owned payload copy before whole-object writes.
- Async listing materially reduces runtime crossings and event-loop wakeups per entry without collecting an unbounded result set.
- Sequential file reads do not add `stat`/`HEAD` unless the requested operation needs the object length.
- Blocking storage I/O does not hold the GIL.
- File cursor, partial-read, buffering, close/durability, cancellation, and free-threaded safety behavior remain unchanged.

Contributor guide

Open the contributing guide

Research direction

Start with the specifically linked Python binding files, such as bindings/python/src/file.rs, operator.rs, lister.rs, and utils.rs, and their referenced tests and benchmark. Pick one work item, reproduce its current behavior with the named coverage, then trace the related core read or writer code before changing it. Done means the selected regression or performance criterion passes without breaking the listed buffering, request, cursor, cancellation, or threading behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.