apache / apache/datafusion

Implement method to apply scalar or aggregate function to Array elements

Open
#15,882 3 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Suppose I have an DataFrame in which one column contains arrays. I wish to be able to apply any scalar expr to each value of that array and return an array out. For example I would like to be able to apply an `abs()` function and convert data such as this:

```
DataFrame()
+--------------+-------------+
| a | abs(a) |
+--------------+-------------+
| [-10, 5, 13] | [10, 5, 13] |
| [2] | [2] |
| [-3, 1] | [3, 1] |
+--------------+-------------+
```

Additionally it would be amazing to be able to apply any aggregate function to an array element.

```
DataFrame()
+--------------+--------+
| a | sum(a) |
+--------------+--------+
| [-10, 5, 13] | 8 |
| [2] | 2 |
| [-3, 1] | 2 |
+--------------+--------+
```

### Describe the solution you'd like

This is similar to the spark `transform` operation. It is very powerful for highly structured data. I don't know the best form that that functions would take, but it would be even more powerful if we could do element-by-element operations across more than one column in the dataframe. There are many use cases where you will have columns of array elements of the same length.

### Describe alternatives you've considered

The current status quo is to either write a UDF to handle these on a case by case basis or to do an unnest and group by. The unnest and group by can be an expensive operation.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

The issue names no implementation files, tests, or entry points. Start by reviewing DataFusion's existing array and scalar or aggregate function support, then compare the requested behavior with Spark's transform operation. Done means defining the function form and supporting element-wise and aggregate operations on array values without requiring unnesting and grouping.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.