apache / apache/datafusion-python

Expose SessionContext.register_higher_order_function / deregister_higher_order_function (v54)

未關閉
#1,572 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
604
分支
174
平均合併
1 天 7 小時
30 天內合併 PR
4

描述

## Background

DataFusion 54 introduced first-class higher-order functions (HOFs) that take lambdas plus collection arguments and rewrite to a scalar `Expr` at planning time. PR #1561 exposed Python lambda syntax and the built-in HOFs (``array_transform``, ``array_filter``, ``array_any_match``, etc.) but did not expose the registration API for custom HOFs.

## Upstream signature

```rust
pub fn register_higher_order_function(&self, func: Arc)
pub fn deregister_higher_order_function(&self, name: &str) -> Option>
```

`HigherOrderFunctionImpl` trait requires `name()`, `args_count()` (min, max), and `invoke(&self, args: &[Expr]) -> Result`.

## User value

Lets library authors add lambda-aware operators that are not in DataFusion's built-in set. Examples: ``array_window(arr, size, x -> ...)``, ``array_partition_by(arr, x -> key(x))``, ``array_zip_with(a, b, (x, y) -> x * y)``, or domain-specific operators on JSON / graph / geo arrays. The HOF runs at logical-plan time, rewriting lambda + args into a standard Expr tree the planner optimizes -- not equivalent to a ScalarUDF.

## Why deferred

Effort estimate is medium-large (~350-550 LOC) and the authoring cost is high for end users: the ``invoke`` callback must return a DataFusion ``Expr`` tree from Python, which requires familiarity with the Expr grammar. Most array-lambda needs are already covered by the built-in HOFs PR #1561 ships. No open user requests at the time of audit. Filed for tracking to complete the v54 HOF surface symmetrically once a concrete use case or extension-library ecosystem emerges.

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

先閱讀 PR #1561,以及針對 HigherOrderFunctionImpl 和 Expr 的上游 SessionContext 註冊簽章。定義用於註冊和取消註冊自訂高階函式的 Python API,包括 invoke 如何回傳一個 Expr;完成標準是 v54 註冊介面已公開並可供函式庫作者使用。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python, rust
領域
api
Issue 類型
功能
難度
5/5
預估耗時
一週以上
活躍度
冷清
描述清晰度
需要釐清
新手友好度
30/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。