Add Native Support for List Functions (Dependent on Lambda Support)
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Working with nested list data requires cumbersome unnesting and re-aggregation steps. Native list functions (list_transform, list_filter, and list_reduce) would let users directly operate on list types with concise, expressive syntax, improving both usability and performance.
This feature relies on support for lambda functions which is tracked in (https://github.com/apache/datafusion/issues/14205).
### Describe the solution you'd like
Implement native list functions that operate directly on nested list types using lambda expressions. For example:
`list_transform`: Apply a lambda function to each element in a list.
`list_filter`: Filter list elements based on a lambda predicate.
`list_reduce`: Aggregate list elements using a lambda function.
Note that the lambda function support itself is not available yet, see #14205 for that work. Once lambdas are supported, these list functions can be built on top of them.
### Describe alternatives you've considered
_No response_
### Additional context
**Example**:
```sql
INSERT INTO foo VALUES
([1], 10),
([1, 2, 3], 5),
([-1, NULL, 2], 7);
SELECT list_transform(col_a, x -> x + col_b) AS transformed_foo
FROM foo;
```
**transformed_foo:**
[11]
[6, 7, 8]
[6, NULL, 9]
Contributor guide
Research direction
Start by reviewing the lambda-function work tracked in issue #14205, since these list functions depend on it. Define completion around native list_transform, list_filter, and list_reduce operations over nested lists with lambda expressions, and verify the SQL example produces the documented transformed values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100