apache / apache/datafusion-python
[Discussion] Define python UDF using `CREATE FUNCTION` SQL statement
- Langage dominant
- Python
- Étoiles
- 604
- Forks
- 174
- Merge moyen
- 1 j 7 h
- PR mergées (30 j)
- 4
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I would like to be able to define a python function using SQL `CREATE FUNCTION` statement.
Something like
```rust
let sql = r#"
CREATE FUNCTION to_miles(DOUBLE)
RETURNS DOUBLE
LANGUAGE PYARROW
AS '
import pyarrow.compute as pc
conversation_rate_multiplier = 0.62137119
def to_miles(km_data):
return pc.multiply(km_data, conversation_rate_multiplier)
'
"#;
ctx.sql(sql).await?.show().await?;
ctx.sql("select double_col, to_miles(double_col) from t")
.await?
.show()
.await?;
```
**Describe the solution you'd like**
datafusion provides pluggable `datafusion::execution::context::FunctionFactory` which could be wired to interpret python function definitions.
we could provide column based function definitions (`LANGUAGE PYARROW`) or row based (`LANGUAGE PYTHON`)
**Describe alternatives you've considered**
**Additional context**
- databricks has [similar functionality](https://docs.databricks.com/aws/en/udf/unity-catalog)
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Piste de recherche
Start at datafusion::execution::context::FunctionFactory and trace how SQL CREATE FUNCTION definitions could be connected to Python UDF execution. Clarify whether column-based LANGUAGE PYARROW and row-based LANGUAGE PYTHON are in scope, then validate the CREATE FUNCTION and SELECT flow shown in the issue.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- python, rust, sql
- Domaine
- databases
- Type d'issue
- Fonctionnalité
- Difficulté
- 5/5
- Temps estimé
- Plus d'une semaine
- Activité
- Active
- Clarté
- À clarifier
- Accessibilité débutants
- 35/100