apache / apache/datafusion

Nested UDF array signatures are unwieldy

Open
#24,141 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

https://github.com/apache/datafusion/blob/70c26a06716eb552f0fc1d2115957a4089504f11/datafusion/expr-common/src/signature.rs#L526-L541

https://github.com/apache/datafusion/blob/70c26a06716eb552f0fc1d2115957a4089504f11/datafusion/expr-common/src/signature.rs#L565-L577

There are two current use cases which aren't well supported by our current array signature code:

- Can't coerce to a specific element type
- Can't handle arrays of different element type

For the first point, see how for `cosine_distance` it needs to define signature via user defined in order to ensure input lists come as `list`:

https://github.com/apache/datafusion/blob/70c26a06716eb552f0fc1d2115957a4089504f11/datafusion/functions-nested/src/cosine_distance.rs#L100-L139

For the second point, an example is `arrays_zip` which can accept lists of different type (though technically since it can accept an arbitrary number of arguments this is also another limitation)

### Describe the solution you'd like

See if we can refine the array signature code so more of our nested UDFs can use it, instead of resorting to user defined

### Describe alternatives you've considered

_No response_

### Additional context

Potentially related:

- https://github.com/apache/datafusion/issues/6559

Contributor guide

Open the contributing guide

Research direction

Start with the array signature code in datafusion/expr-common/src/signature.rs at the referenced sections, then compare the user-defined signature in datafusion/functions-nested/src/cosine_distance.rs and the arrays_zip implementation. Determine what the existing signature model cannot express. Done means nested UDFs can express fixed element coercion and differing element types without resorting to user-defined signatures.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.