influxdata / influxdata/datafusion-udf-wasm

Support Not-NULL Python UDFs

Open
#69 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
20
Forks
3
Avg merge
6h 20m
Merged PRs (30d)
24

Description

# What
Currently always treat Python UDF return values as "nullable", however that doesn't have to be the case.

# Why
This helps DataFusion to optimize things further.

# How
## WASM UDF
Our WASM UDF interface currently only exposes the [return type](https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#tymethod.return_type), but not the [return field](https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.return_field_from_args). We would need to wire that up first to support non-nullable return values.

## Python Inference
There are two cases where we can easily infer non-nullable results:

### Python Method w/ Full NULL Handling
Something like:

```python
def add(a: int | None, b: int | None) -> int
if a is None:
a = 0
if b is None:
b = 0
return a + b
```

The user handles the nullability of the arguments and promises to never return NULL.

### Python Method w/ Auto-NULL Handling
Something like:

```python
def add(a: int, b: int) -> int
return a + b
```

The user opted to let us do the NULL handling, i.e. `add` is never called if either of the two inputs in NULL. However if we the two input fields are non-nullable, then we can be sure that the output is also never NULL because we would never skip calling the Python method.

Contributor guide

Open the contributing guide

Research direction

Start with the WASM UDF interface's return_type and return_field_from_args entry points, then trace the Python UDF null-handling and type-inference paths. Done means Python UDFs can expose or infer non-nullable return values, including the full-NULL-handling and auto-NULL-handling cases, and DataFusion receives that nullability.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust, wasm
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.