apache / apache/datafusion-python
[Discussion] Define python UDF using `CREATE FUNCTION` SQL statement
- 主要語言
- Python
- 星號
- 604
- 分支
- 174
- 平均合併
- 2 天 22 小時
- 30 天內合併 PR
- 5
描述
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I would like to be able to define a python function using SQL `CREATE FUNCTION` statement.
Something like
```rust
let sql = r#"
CREATE FUNCTION to_miles(DOUBLE)
RETURNS DOUBLE
LANGUAGE PYARROW
AS '
import pyarrow.compute as pc
conversation_rate_multiplier = 0.62137119
def to_miles(km_data):
return pc.multiply(km_data, conversation_rate_multiplier)
'
"#;
ctx.sql(sql).await?.show().await?;
ctx.sql("select double_col, to_miles(double_col) from t")
.await?
.show()
.await?;
```
**Describe the solution you'd like**
datafusion provides pluggable `datafusion::execution::context::FunctionFactory` which could be wired to interpret python function definitions.
we could provide column based function definitions (`LANGUAGE PYARROW`) or row based (`LANGUAGE PYTHON`)
**Describe alternatives you've considered**
**Additional context**
- databricks has [similar functionality](https://docs.databricks.com/aws/en/udf/unity-catalog)
貢獻指南
這個儲存庫沒有索引到貢獻指南
研究方向
從 datafusion::execution::context::FunctionFactory 開始,追蹤如何將 SQL CREATE FUNCTION 定義與 Python UDF 執行連接起來。釐清以欄為基礎的 LANGUAGE PYARROW 和以列為基礎的 LANGUAGE PYTHON 是否在範圍內,然後驗證 issue 中所示的 CREATE FUNCTION 和 SELECT 流程。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- python, rust, sql
- 領域
- databases
- Issue 類型
- 功能
- 難度
- 5/5
- 預估耗時
- 一週以上
- 活躍度
- 活躍
- 描述清晰度
- 需要釐清
- 新手友好度
- 35/100