apache / apache/datafusion-python

Expose Python UDF inlining as a public, reusable extension codec

Aperta
#1,704 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
enhancement rust
Lingua principale
Python
Stelle
604
Fork
174
Merge medio
1g 7h
PR unite (30g)
4

Descrizione

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

datafusion-python can serialize a Python scalar UDF inline, by cloudpickling the function together with its schemas and volatility, so a plan referencing a Python UDF round-trips into a process that does not have that function registered. This is what `SessionContext.with_python_udf_inlining` turns on.

That capability is not reachable from outside the crate. `PythonFunctionScalarUDF` is `pub(crate)`, and `encode_python_scalar_udf` / `decode_python_scalar_udf` in `crates/core/src/codec.rs` are private; only the `try_encode_python_scalar_udf` / `try_decode_python_scalar_udf` wrappers are `pub(crate)`, and they are called from the two codecs this crate installs on every session. A Rust project that already has its own physical extension codec, and wants Python UDFs to travel through it, has to reimplement the cloudpickle framing to stay wire-compatible. Requested in https://github.com/apache/datafusion-python/pull/1678#pullrequestreview-5100366976.

**Describe the solution you'd like**

Extract the inlining behavior into its own `LogicalExtensionCodec` / `PhysicalExtensionCodec` implementation and make that type public, rather than exporting two free functions and a struct. Since #1678 made codecs compose, `with_python_udf_inlining` can then install that codec into the chain instead of being a flag consulted inside `PythonLogicalCodec` and `PythonPhysicalCodec`, and an outside consumer can put the same type in its own chain and get the same wire format for free. Suggested independently in https://github.com/apache/datafusion-python/pull/1678#discussion_r3918287124.

A public codec is a better thing to commit to than the current internals. It hides the payload layout, which is versioned and should stay free to change, and it exposes the behavior at the granularity DataFusion already understands.

**Describe alternatives you've considered**

Marking the existing functions and `PythonFunctionScalarUDF` `pub` as they stand. Smaller, but it publishes the payload layout as API, and it does not compose with anything — a caller still has to decide where in its own codec to call them.

**Additional context**

Depends on #1703, gating `pyo3/extension-module`, since the crate cannot currently be linked as a plain Rust dependency. Follow-up from #1678.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Start with crates/core/src/codec.rs and the PythonLogicalCodec, PythonPhysicalCodec, and SessionContext.with_python_udf_inlining entry points. Read dependency #1703 and the codec composition changes from #1678 before determining how the public codec should be installed in the chain. Done means external Rust consumers can add the public codec and preserve the existing Python UDF wire format without reimplementing its payload framing.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, rust
Ambito
backend-api-design
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Attiva
Chiarezza
Abbastanza chiara
Idoneità per principianti
48/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.