influxdata / influxdata/datafusion-udf-wasm
Support Not-NULL Python UDFs
- Dominant language
- Rust
- Stars
- 20
- Forks
- 3
- Avg merge
- 6h 20m
- Merged PRs (30d)
- 24
Description
# What
Currently always treat Python UDF return values as "nullable", however that doesn't have to be the case.
# Why
This helps DataFusion to optimize things further.
# How
## WASM UDF
Our WASM UDF interface currently only exposes the [return type](https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#tymethod.return_type), but not the [return field](https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.return_field_from_args). We would need to wire that up first to support non-nullable return values.
## Python Inference
There are two cases where we can easily infer non-nullable results:
### Python Method w/ Full NULL Handling
Something like:
```python
def add(a: int | None, b: int | None) -> int
if a is None:
a = 0
if b is None:
b = 0
return a + b
```
The user handles the nullability of the arguments and promises to never return NULL.
### Python Method w/ Auto-NULL Handling
Something like:
```python
def add(a: int, b: int) -> int
return a + b
```
The user opted to let us do the NULL handling, i.e. `add` is never called if either of the two inputs in NULL. However if we the two input fields are non-nullable, then we can be sure that the output is also never NULL because we would never skip calling the Python method.
Contributor guide
Research direction
Start with the WASM UDF interface's return_type and return_field_from_args entry points, then trace the Python UDF null-handling and type-inference paths. Done means Python UDFs can expose or infer non-nullable return values, including the full-NULL-handling and auto-NULL-handling cases, and DataFusion receives that nullability.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust, wasm
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100