apache / apache/datafusion-python
Expose Python UDF inlining as a public, reusable extension codec
- 主要語言
- Python
- 星號
- 604
- 分支
- 174
- 平均合併
- 1 天 7 小時
- 30 天內合併 PR
- 4
描述
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
datafusion-python can serialize a Python scalar UDF inline, by cloudpickling the function together with its schemas and volatility, so a plan referencing a Python UDF round-trips into a process that does not have that function registered. This is what `SessionContext.with_python_udf_inlining` turns on.
That capability is not reachable from outside the crate. `PythonFunctionScalarUDF` is `pub(crate)`, and `encode_python_scalar_udf` / `decode_python_scalar_udf` in `crates/core/src/codec.rs` are private; only the `try_encode_python_scalar_udf` / `try_decode_python_scalar_udf` wrappers are `pub(crate)`, and they are called from the two codecs this crate installs on every session. A Rust project that already has its own physical extension codec, and wants Python UDFs to travel through it, has to reimplement the cloudpickle framing to stay wire-compatible. Requested in https://github.com/apache/datafusion-python/pull/1678#pullrequestreview-5100366976.
**Describe the solution you'd like**
Extract the inlining behavior into its own `LogicalExtensionCodec` / `PhysicalExtensionCodec` implementation and make that type public, rather than exporting two free functions and a struct. Since #1678 made codecs compose, `with_python_udf_inlining` can then install that codec into the chain instead of being a flag consulted inside `PythonLogicalCodec` and `PythonPhysicalCodec`, and an outside consumer can put the same type in its own chain and get the same wire format for free. Suggested independently in https://github.com/apache/datafusion-python/pull/1678#discussion_r3918287124.
A public codec is a better thing to commit to than the current internals. It hides the payload layout, which is versioned and should stay free to change, and it exposes the behavior at the granularity DataFusion already understands.
**Describe alternatives you've considered**
Marking the existing functions and `PythonFunctionScalarUDF` `pub` as they stand. Smaller, but it publishes the payload layout as API, and it does not compose with anything — a caller still has to decide where in its own codec to call them.
**Additional context**
Depends on #1703, gating `pyo3/extension-module`, since the crate cannot currently be linked as a plain Rust dependency. Follow-up from #1678.
貢獻指南
這個儲存庫沒有索引到貢獻指南
研究方向
從 crates/core/src/codec.rs 以及 PythonLogicalCodec、PythonPhysicalCodec 和 SessionContext.with_python_udf_inlining 入口開始。在確定應如何將公開 codec 安裝到鏈中之前,先閱讀依賴 #1703 和 #1678 中的 codec 組合變更。完成的標準是,外部 Rust 消費者可以加入公開 codec,並保留現有的 Python UDF wire format,而無需重新實作其 payload framing。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- python, rust
- 領域
- backend-api-design
- Issue 類型
- 功能
- 難度
- 5/5
- 預估耗時
- 一週以上
- 活躍度
- 活躍
- 描述清晰度
- 基本清楚
- 新手友好度
- 48/100