Tracking issue: Python binding in multiprocessing context
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
The operator is stateless, so we have no reason to reject its use in Python's multiprocess environment.
### Problem and Solution
Python uses the pickle format to serialize and deserialize Python objects between multiple processes. If it is a native Python object, it is naturally supported to be converted by pickle. However, as an extension, we must implement it manually.
There are two hooks for the purpose, `__setstate__` and `__getstate__`.
However, the two methods means that a Operator must expose all information that can be used to construct it. Currently, the public interface can't do that.
@Zheaoli Can you provide some ideas to achieve the purpose?
### Additional Context
Here are examples in polars:
https://github.com/pola-rs/polars/blob/18786acd8d1eb68fc87982b07ce29ecbae0923f0/crates/polars-python/src/lazyframe/serde.rs#L16-L36
### Are you willing to contribute to the development of this feature?
- [X] Yes, I am willing to contribute to the development of this feature.
Contributor guide
Research direction
Start by reading the public Operator interface and the Python binding around the __setstate__ and __getstate__ hooks, then compare the linked polars serde.rs example. Done means defining how an Operator exposes enough state to be serialized and reconstructed safely across Python multiprocessing contexts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- api, developer-experience
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100