apache / apache/datafusion-python

Expose Python UDF inlining as a public, reusable extension codec

Offen
#1,704 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
enhancement rust
Vorherrschende Sprache
Python
Sterne
604
Forks
174
Ø Merge
1 T. 7 Std.
Gemergte PRs (30 T.)
4

Beschreibung

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

datafusion-python can serialize a Python scalar UDF inline, by cloudpickling the function together with its schemas and volatility, so a plan referencing a Python UDF round-trips into a process that does not have that function registered. This is what `SessionContext.with_python_udf_inlining` turns on.

That capability is not reachable from outside the crate. `PythonFunctionScalarUDF` is `pub(crate)`, and `encode_python_scalar_udf` / `decode_python_scalar_udf` in `crates/core/src/codec.rs` are private; only the `try_encode_python_scalar_udf` / `try_decode_python_scalar_udf` wrappers are `pub(crate)`, and they are called from the two codecs this crate installs on every session. A Rust project that already has its own physical extension codec, and wants Python UDFs to travel through it, has to reimplement the cloudpickle framing to stay wire-compatible. Requested in https://github.com/apache/datafusion-python/pull/1678#pullrequestreview-5100366976.

**Describe the solution you'd like**

Extract the inlining behavior into its own `LogicalExtensionCodec` / `PhysicalExtensionCodec` implementation and make that type public, rather than exporting two free functions and a struct. Since #1678 made codecs compose, `with_python_udf_inlining` can then install that codec into the chain instead of being a flag consulted inside `PythonLogicalCodec` and `PythonPhysicalCodec`, and an outside consumer can put the same type in its own chain and get the same wire format for free. Suggested independently in https://github.com/apache/datafusion-python/pull/1678#discussion_r3918287124.

A public codec is a better thing to commit to than the current internals. It hides the payload layout, which is versioned and should stay free to change, and it exposes the behavior at the granularity DataFusion already understands.

**Describe alternatives you've considered**

Marking the existing functions and `PythonFunctionScalarUDF` `pub` as they stand. Smaller, but it publishes the payload layout as API, and it does not compose with anything — a caller still has to decide where in its own codec to call them.

**Additional context**

Depends on #1703, gating `pyo3/extension-module`, since the crate cannot currently be linked as a plain Rust dependency. Follow-up from #1678.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Start with crates/core/src/codec.rs and the PythonLogicalCodec, PythonPhysicalCodec, and SessionContext.with_python_udf_inlining entry points. Read dependency #1703 and the codec composition changes from #1678 before determining how the public codec should be installed in the chain. Done means external Rust consumers can add the public codec and preserve the existing Python UDF wire format without reimplementing its payload framing.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python, rust
Bereich
backend-api-design
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Aktiv
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
48/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.