apache / apache/datafusion-ballista

Add support for Python UDFs in distributed queries

Open
#173 23 comments 5 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I would like to be able to execute queries containing custom Python code. This is already supported in DataFusion but we need to add the serde aspect in Ballista.

**Describe the solution you'd like**
It looks like we can serialize Python functions with https://docs.rs/pyo3/0.7.0/pyo3/marshal/index.html

We then need to store the serialied Python functions in protobuf and deserialize them in the executors.

**Describe alternatives you've considered**
None

**Additional context**
None

Contributor guide

Open the contributing guide

Research direction

Start by reviewing DataFusion's existing Python UDF support and the PyO3 marshal approach, then trace how Ballista stores query information in protobuf and hands it to executors. Done means Python functions can be serialized into protobuf and deserialized by executors when running distributed queries.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.