apache / apache/datafusion-python

[Discussion] Define python UDF using `CREATE FUNCTION` SQL statement

Ouverte
#1,701 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
enhancement
Langage dominant
Python
Étoiles
604
Forks
174
Merge moyen
1 j 7 h
PR mergées (30 j)
4

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

I would like to be able to define a python function using SQL `CREATE FUNCTION` statement.

Something like

```rust
let sql = r#"
CREATE FUNCTION to_miles(DOUBLE)
RETURNS DOUBLE
LANGUAGE PYARROW
AS '
import pyarrow.compute as pc

conversation_rate_multiplier = 0.62137119

def to_miles(km_data):
return pc.multiply(km_data, conversation_rate_multiplier)
'
"#;

ctx.sql(sql).await?.show().await?;

ctx.sql("select double_col, to_miles(double_col) from t")
.await?
.show()
.await?;
```

**Describe the solution you'd like**

datafusion provides pluggable `datafusion::execution::context::FunctionFactory` which could be wired to interpret python function definitions.

we could provide column based function definitions (`LANGUAGE PYARROW`) or row based (`LANGUAGE PYTHON`)

**Describe alternatives you've considered**

**Additional context**

- databricks has [similar functionality](https://docs.databricks.com/aws/en/udf/unity-catalog)

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Start at datafusion::execution::context::FunctionFactory and trace how SQL CREATE FUNCTION definitions could be connected to Python UDF execution. Clarify whether column-based LANGUAGE PYARROW and row-based LANGUAGE PYTHON are in scope, then validate the CREATE FUNCTION and SELECT flow shown in the issue.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python, rust, sql
Domaine
databases
Type d'issue
Fonctionnalité
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
Active
Clarté
À clarifier
Accessibilité débutants
35/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.