apache / apache/datafusion

Expose inner field of struct within list-array

Open
#11,419 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

We frequently work with tables made up of "batch" data, which is in turn represented via structs.

For example:
```
+--------------------------------------+
| position2d |
+--------------------------------------+
| [{x: 1.0, y: 2.0}, {x: 3.0, y: 4.0}] |
| [{x: 5.0, y: 6.0}] |
+--------------------------------------+
```

I want to be able to restructure this so the inner fields of the struct array become their own columns:
```
+------------+------------+
| X | Y |
+------------+------------+
| [1.0, 3.0] | [2.0, 4.0] |
| [5.0] | [6.0] |
+------------+------------+
```

### Describe the solution you'd like

I would like to be able to do this from SQL.

For example:
```
SELECT array_field(position2d, "x"), array_field(position2d, "Y") FROM example;
```

### Describe alternatives you've considered

This can be achieved via unnest and array_agg, but is somewhat painful to do so, requires the existence of a preserved row_id for group_by operation, and introduces uncertainty as to preservation of ordering. It does not appear that datafusion supports `WITH ORDINALITY` which would be used to orderwise guarantee ordering is maintained.

Example:
```
CREATE TABLE example AS
SELECT * FROM (
VALUES
(1, ARRAY[NAMED_STRUCT('x', 1.0, 'y', 2.0),
NAMED_STRUCT('x', 3.0, 'y', 4.0)]),
(2, ARRAY[NAMED_STRUCT('x', 5.0, 'y', 6.0)])
) as example(id, position2d);

select array_agg(p.x) as x, array_agg(p.y) as y
FROM (select id, unnest(position2d) as p from example)
GROUP BY id ORDER BY id;
```

### Additional context

Structurally, the appropriate child array of the struct should be able to be used with the offset array from the list-array and I believe the "right thing" should happen. As such I believe this should generally be able to be implemented as a cheap operation along the lines of a cast.

Contributor guide

Open the contributing guide

Research direction

Start with the SQL examples using array_field, unnest, and array_agg, then trace how list arrays and struct fields are represented and projected. Done means SQL can expose each inner struct field as an array while preserving the list offsets and element ordering, with coverage for the examples shown in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.