apache / apache/datafusion-python
[Discussion] Define python UDF using `CREATE FUNCTION` SQL statement
- Lingua principale
- Python
- Stelle
- 604
- Fork
- 174
- Merge medio
- 1g 7h
- PR unite (30g)
- 4
Descrizione
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I would like to be able to define a python function using SQL `CREATE FUNCTION` statement.
Something like
```rust
let sql = r#"
CREATE FUNCTION to_miles(DOUBLE)
RETURNS DOUBLE
LANGUAGE PYARROW
AS '
import pyarrow.compute as pc
conversation_rate_multiplier = 0.62137119
def to_miles(km_data):
return pc.multiply(km_data, conversation_rate_multiplier)
'
"#;
ctx.sql(sql).await?.show().await?;
ctx.sql("select double_col, to_miles(double_col) from t")
.await?
.show()
.await?;
```
**Describe the solution you'd like**
datafusion provides pluggable `datafusion::execution::context::FunctionFactory` which could be wired to interpret python function definitions.
we could provide column based function definitions (`LANGUAGE PYARROW`) or row based (`LANGUAGE PYTHON`)
**Describe alternatives you've considered**
**Additional context**
- databricks has [similar functionality](https://docs.databricks.com/aws/en/udf/unity-catalog)
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Start at datafusion::execution::context::FunctionFactory and trace how SQL CREATE FUNCTION definitions could be connected to Python UDF execution. Clarify whether column-based LANGUAGE PYARROW and row-based LANGUAGE PYTHON are in scope, then validate the CREATE FUNCTION and SELECT flow shown in the issue.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python, rust, sql
- Ambito
- databases
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Attiva
- Chiarezza
- Da chiarire
- Idoneità per principianti
- 35/100