apache / apache/datafusion-ballista
Add support for Python UDFs in distributed queries
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I would like to be able to execute queries containing custom Python code. This is already supported in DataFusion but we need to add the serde aspect in Ballista.
**Describe the solution you'd like**
It looks like we can serialize Python functions with https://docs.rs/pyo3/0.7.0/pyo3/marshal/index.html
We then need to store the serialied Python functions in protobuf and deserialize them in the executors.
**Describe alternatives you've considered**
None
**Additional context**
None
Contributor guide
Research direction
Start by reviewing DataFusion's existing Python UDF support and the PyO3 marshal approach, then trace how Ballista stores query information in protobuf and hands it to executors. Done means Python functions can be serialized into protobuf and deserialized by executors when running distributed queries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100