apache / apache/datafusion-python
[Discussion] Define python UDF using `CREATE FUNCTION` SQL statement
- Dominant language
- Python
- Stars
- 604
- Forks
- 174
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 4
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I would like to be able to define a python function using SQL `CREATE FUNCTION` statement.
Something like
```rust
let sql = r#"
CREATE FUNCTION to_miles(DOUBLE)
RETURNS DOUBLE
LANGUAGE PYARROW
AS '
import pyarrow.compute as pc
conversation_rate_multiplier = 0.62137119
def to_miles(km_data):
return pc.multiply(km_data, conversation_rate_multiplier)
'
"#;
ctx.sql(sql).await?.show().await?;
ctx.sql("select double_col, to_miles(double_col) from t")
.await?
.show()
.await?;
```
**Describe the solution you'd like**
datafusion provides pluggable `datafusion::execution::context::FunctionFactory` which could be wired to interpret python function definitions.
we could provide column based function definitions (`LANGUAGE PYARROW`) or row based (`LANGUAGE PYTHON`)
**Describe alternatives you've considered**
**Additional context**
- databricks has [similar functionality](https://docs.databricks.com/aws/en/udf/unity-catalog)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at datafusion::execution::context::FunctionFactory and trace how SQL CREATE FUNCTION definitions could be connected to Python UDF execution. Clarify whether column-based LANGUAGE PYARROW and row-based LANGUAGE PYTHON are in scope, then validate the CREATE FUNCTION and SELECT flow shown in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100