apache / apache/datafusion

Simple Functions

Open
#12,635 24 comments 11 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Design doc

- https://docs.google.com/document/d/1JyfbYfpmjvd3N3LEVAN7tX1D9ze6N98yGx3g4po_WoQ

### Is your feature request related to a problem or challenge?

#### Verbosity

Currently implementing a scalar function is a pretty involved process. For example a simple function calculating greatest common divisor looks like this (a lot of detail omitted for brevity):

```rust
impl ScalarUDFImpl for GcdFunc {
fn name(&self) -> &str {
"gcd"
}

fn return_type(&self, _arg_types: &[DataType]) -> Result {
Ok(Int64)
}

fn invoke(&self, args: &[ColumnarValue]) -> Result {
make_scalar_function(gcd, vec![])(args)
}
}

/// Gcd SQL function
fn gcd(args: &[ArrayRef]) -> Result {
match args[0].data_type() {
Int64 => {
let arg1 = downcast_arg!(&args[0], "x", Int64Array);
let arg2 = downcast_arg!(&args[1], "y", Int64Array);

Ok(arg1
.iter()
.zip(arg2.iter())
.map(|(a1, a2)| match (a1, a2) {
(Some(a1), Some(a2)) => Ok(Some(compute_gcd(a1, a2)?)),
_ => Ok(None),
})
.collect::>()
.map(Arc::new)? as ArrayRef)
}
other => exec_err!("Unsupported data type {other:?} for function gcd"),
}
}

/// Computes greatest common divisor using Binary GCD algorithm.
pub fn compute_gcd(x: i64, y: i64) -> Result {
...
}
```

This function is still "relatively simple" because:

- it handles only int64 type (has no overloads)
- a more complicated function would calculate return type based in input types in `return_type` and again in `invoke`
- doesn't not support constant `x` or constant `y` parameters
- the `make_scalar_function` helper expands scalar constant argument into an array, thus `gcd(x, 5)` is less efficient than `gcd(x, y)` because it first needs to allocate a temporary buffer for `5`s repeated times the batch length
- it doesn't handle dictionaries or run-end (REE / RLE) encoded values
- i doubt people frequently calculate `gcd` on partitioning keys, but this is just an example
- it doesn't handle all-non-null case, so probably misses optimization/vectorization opportunity

It should be possible to express functions using simple row-level operations, because in query engines most function logic is structured like that anyway (`compute_gcd` in the example).

#### Local data structures

Currently DataFusion functions are singletons plugged into the execution engine. They have no way to store and reuse buffers or compiled regular expressions, etc.
Thread local storage is not an option because DataFusion, when embedded, does not control creation of application threads.

### Describe the solution you'd like

#### Simple

It should be possible to separate function logic from the mechanics.
Exemplary GCD function needs to provide `fn compute_gcd(x: i64, y: i64) -> Result` and the rest of boilerplate should not be hand-written for every function separately.

It should be possible to implement a function that accepts string values, without having to deal with the 5 different runtime representations that can carry string values: `StringArray, LargeStringArray, StringViewArray, DictionaryArray, RunArray` (maybe more than 5 because they are recursive in theory: can `DictionaryArray` contain a `DictionaryArray`? can it contain `RunArray`?)

#### Focused

Because SQL is statically typed, it is necessary to select function overload during planning time, so that return type is also known (the `return_type` function). The `invoke` function needs to implement same logic again. This process should be less error-prone: once the function is bound to the query call site, its signature is known and the implementation should not need to do type checks.

#### Performant / Efficient

It should be the compiler's / framework's job to provide vectorization, without hand-writing same logic in every function separately.

It should be the compiler's / framework's to do null checks, providing efficient tight loops for the all-non-null case without having to write such logic in every function separately.

It should be possible to write efficient functions that need thread local data structures, for example for regular expressions, without having to use thread locals and/or shared pools which introduce contention.

### Describe alternatives you've considered

- https://github.com/apache/datafusion/issues/11413

However, direct use of the library is not possible because
- it operates on Arrow arrays directly, doesn't fit DataFusion execution model
- it operates on Arrow data types, but DataFusion needs its own type system: https://github.com/apache/datafusion/issues/11513
- The problem space for a query engine is much more complicated

The library could still be part of the solution, but doesn't have to be and it's a non-goal to use a particular library.

### Additional context

- https://github.com/apache/datafusion/issues/11513
- https://github.com/apache/datafusion/discussions/7421
- How function definitions look like in Trino, for example a function operating on Int64-equivalent https://github.com/trinodb/trino/blob/e6aabfec55fcf5e4dd3c283b9a29da97c1301f6c/core/trino-main/src/main/java/io/trino/operator/scalar/MathFunctions.java#L128-L135
- Velox "Simple Functions" https://vldb.org/pvldb/vol15/p3372-pedreira.pdf

Contributor guide

Open the contributing guide

Research direction

Start with the linked design document and the existing ScalarUDFImpl, invoke, return_type, and make_scalar_function patterns shown in the issue. Review related issues 11413 and 11513 plus the cited DataFusion discussion and external examples; done requires an agreed design for simpler, typed, efficient functions and reusable local data structures.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.