apache / apache/opendal

Tracking issue: Python binding in multiprocessing context

Open
#5,316 4 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

The operator is stateless, so we have no reason to reject its use in Python's multiprocess environment.

### Problem and Solution

Python uses the pickle format to serialize and deserialize Python objects between multiple processes. If it is a native Python object, it is naturally supported to be converted by pickle. However, as an extension, we must implement it manually.

There are two hooks for the purpose, `__setstate__` and `__getstate__`.

However, the two methods means that a Operator must expose all information that can be used to construct it. Currently, the public interface can't do that.

@Zheaoli Can you provide some ideas to achieve the purpose?

### Additional Context

Here are examples in polars:

https://github.com/pola-rs/polars/blob/18786acd8d1eb68fc87982b07ce29ecbae0923f0/crates/polars-python/src/lazyframe/serde.rs#L16-L36

### Are you willing to contribute to the development of this feature?

- [X] Yes, I am willing to contribute to the development of this feature.

Contributor guide

Open the contributing guide

Research direction

Start by reading the public Operator interface and the Python binding around the __setstate__ and __getstate__ hooks, then compare the linked polars serde.rs example. Done means defining how an Operator exposes enough state to be serialized and reconstructed safely across Python multiprocessing contexts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
api, developer-experience
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.