apache / apache/datafusion-python

Expose Python UDF inlining as a public, reusable extension codec

Abierto
#1,704 0 comentarios 0 reacciones 0 asignados Ver en GitHub
enhancement rust
Lenguaje dominante
Python
Estrellas
604
Forks
174
Merge medio
1 d 7 h
PR fusionados (30 d)
4

Descripción

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

datafusion-python can serialize a Python scalar UDF inline, by cloudpickling the function together with its schemas and volatility, so a plan referencing a Python UDF round-trips into a process that does not have that function registered. This is what `SessionContext.with_python_udf_inlining` turns on.

That capability is not reachable from outside the crate. `PythonFunctionScalarUDF` is `pub(crate)`, and `encode_python_scalar_udf` / `decode_python_scalar_udf` in `crates/core/src/codec.rs` are private; only the `try_encode_python_scalar_udf` / `try_decode_python_scalar_udf` wrappers are `pub(crate)`, and they are called from the two codecs this crate installs on every session. A Rust project that already has its own physical extension codec, and wants Python UDFs to travel through it, has to reimplement the cloudpickle framing to stay wire-compatible. Requested in https://github.com/apache/datafusion-python/pull/1678#pullrequestreview-5100366976.

**Describe the solution you'd like**

Extract the inlining behavior into its own `LogicalExtensionCodec` / `PhysicalExtensionCodec` implementation and make that type public, rather than exporting two free functions and a struct. Since #1678 made codecs compose, `with_python_udf_inlining` can then install that codec into the chain instead of being a flag consulted inside `PythonLogicalCodec` and `PythonPhysicalCodec`, and an outside consumer can put the same type in its own chain and get the same wire format for free. Suggested independently in https://github.com/apache/datafusion-python/pull/1678#discussion_r3918287124.

A public codec is a better thing to commit to than the current internals. It hides the payload layout, which is versioned and should stay free to change, and it exposes the behavior at the granularity DataFusion already understands.

**Describe alternatives you've considered**

Marking the existing functions and `PythonFunctionScalarUDF` `pub` as they stand. Smaller, but it publishes the payload layout as API, and it does not compose with anything — a caller still has to decide where in its own codec to call them.

**Additional context**

Depends on #1703, gating `pyo3/extension-module`, since the crate cannot currently be linked as a plain Rust dependency. Follow-up from #1678.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Start with crates/core/src/codec.rs and the PythonLogicalCodec, PythonPhysicalCodec, and SessionContext.with_python_udf_inlining entry points. Read dependency #1703 and the codec composition changes from #1678 before determining how the public codec should be installed in the chain. Done means external Rust consumers can add the public codec and preserve the existing Python UDF wire format without reimplementing its payload framing.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python, rust
Área
backend-api-design
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Activo
Claridad
Bastante claro
Aptitud para principiantes
48/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.