apache / apache/datafusion-python

[Discussion] Define python UDF using `CREATE FUNCTION` SQL statement

Aperta
#1,701 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
Python
Stelle
604
Fork
174
Merge medio
1g 7h
PR unite (30g)
4

Descrizione

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

I would like to be able to define a python function using SQL `CREATE FUNCTION` statement.

Something like

```rust
let sql = r#"
CREATE FUNCTION to_miles(DOUBLE)
RETURNS DOUBLE
LANGUAGE PYARROW
AS '
import pyarrow.compute as pc

conversation_rate_multiplier = 0.62137119

def to_miles(km_data):
return pc.multiply(km_data, conversation_rate_multiplier)
'
"#;

ctx.sql(sql).await?.show().await?;

ctx.sql("select double_col, to_miles(double_col) from t")
.await?
.show()
.await?;
```

**Describe the solution you'd like**

datafusion provides pluggable `datafusion::execution::context::FunctionFactory` which could be wired to interpret python function definitions.

we could provide column based function definitions (`LANGUAGE PYARROW`) or row based (`LANGUAGE PYTHON`)

**Describe alternatives you've considered**

**Additional context**

- databricks has [similar functionality](https://docs.databricks.com/aws/en/udf/unity-catalog)

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Start at datafusion::execution::context::FunctionFactory and trace how SQL CREATE FUNCTION definitions could be connected to Python UDF execution. Clarify whether column-based LANGUAGE PYARROW and row-based LANGUAGE PYTHON are in scope, then validate the CREATE FUNCTION and SELECT flow shown in the issue.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, rust, sql
Ambito
databases
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Attiva
Chiarezza
Da chiarire
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.