apache / apache/datafusion

Add Native Support for List Functions (Dependent on Lambda Support)

Open
#14,509 2 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Working with nested list data requires cumbersome unnesting and re-aggregation steps. Native list functions (list_transform, list_filter, and list_reduce) would let users directly operate on list types with concise, expressive syntax, improving both usability and performance.

This feature relies on support for lambda functions which is tracked in (https://github.com/apache/datafusion/issues/14205).

### Describe the solution you'd like

Implement native list functions that operate directly on nested list types using lambda expressions. For example:

`list_transform`: Apply a lambda function to each element in a list.
`list_filter`: Filter list elements based on a lambda predicate.
`list_reduce`: Aggregate list elements using a lambda function.

Note that the lambda function support itself is not available yet, see #14205 for that work. Once lambdas are supported, these list functions can be built on top of them.

### Describe alternatives you've considered

_No response_

### Additional context

**Example**:

```sql
INSERT INTO foo VALUES
([1], 10),
([1, 2, 3], 5),
([-1, NULL, 2], 7);

SELECT list_transform(col_a, x -> x + col_b) AS transformed_foo
FROM foo;
```

**transformed_foo:**
[11]
[6, 7, 8]
[6, NULL, 9]

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the lambda-function work tracked in issue #14205, since these list functions depend on it. Define completion around native list_transform, list_filter, and list_reduce operations over nested lists with lambda expressions, and verify the SQL example produces the documented transformed values.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.