apache / apache/datafusion-python

Expose Python UDF inlining as a public, reusable extension codec

Open
#1,704 0 comments 0 reactions 0 assignees View on GitHub
enhancement rust
Dominant language
Python
Stars
604
Forks
174
Avg merge
1d 7h
Merged PRs (30d)
4

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

datafusion-python can serialize a Python scalar UDF inline, by cloudpickling the function together with its schemas and volatility, so a plan referencing a Python UDF round-trips into a process that does not have that function registered. This is what `SessionContext.with_python_udf_inlining` turns on.

That capability is not reachable from outside the crate. `PythonFunctionScalarUDF` is `pub(crate)`, and `encode_python_scalar_udf` / `decode_python_scalar_udf` in `crates/core/src/codec.rs` are private; only the `try_encode_python_scalar_udf` / `try_decode_python_scalar_udf` wrappers are `pub(crate)`, and they are called from the two codecs this crate installs on every session. A Rust project that already has its own physical extension codec, and wants Python UDFs to travel through it, has to reimplement the cloudpickle framing to stay wire-compatible. Requested in https://github.com/apache/datafusion-python/pull/1678#pullrequestreview-5100366976.

**Describe the solution you'd like**

Extract the inlining behavior into its own `LogicalExtensionCodec` / `PhysicalExtensionCodec` implementation and make that type public, rather than exporting two free functions and a struct. Since #1678 made codecs compose, `with_python_udf_inlining` can then install that codec into the chain instead of being a flag consulted inside `PythonLogicalCodec` and `PythonPhysicalCodec`, and an outside consumer can put the same type in its own chain and get the same wire format for free. Suggested independently in https://github.com/apache/datafusion-python/pull/1678#discussion_r3918287124.

A public codec is a better thing to commit to than the current internals. It hides the payload layout, which is versioned and should stay free to change, and it exposes the behavior at the granularity DataFusion already understands.

**Describe alternatives you've considered**

Marking the existing functions and `PythonFunctionScalarUDF` `pub` as they stand. Smaller, but it publishes the payload layout as API, and it does not compose with anything — a caller still has to decide where in its own codec to call them.

**Additional context**

Depends on #1703, gating `pyo3/extension-module`, since the crate cannot currently be linked as a plain Rust dependency. Follow-up from #1678.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with crates/core/src/codec.rs and the PythonLogicalCodec, PythonPhysicalCodec, and SessionContext.with_python_udf_inlining entry points. Read dependency #1703 and the codec composition changes from #1678 before determining how the public codec should be installed in the chain. Done means external Rust consumers can add the public codec and preserve the existing Python UDF wire format without reimplementing its payload framing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.