Expose inner field of struct within list-array
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
We frequently work with tables made up of "batch" data, which is in turn represented via structs.
For example:
```
+--------------------------------------+
| position2d |
+--------------------------------------+
| [{x: 1.0, y: 2.0}, {x: 3.0, y: 4.0}] |
| [{x: 5.0, y: 6.0}] |
+--------------------------------------+
```
I want to be able to restructure this so the inner fields of the struct array become their own columns:
```
+------------+------------+
| X | Y |
+------------+------------+
| [1.0, 3.0] | [2.0, 4.0] |
| [5.0] | [6.0] |
+------------+------------+
```
### Describe the solution you'd like
I would like to be able to do this from SQL.
For example:
```
SELECT array_field(position2d, "x"), array_field(position2d, "Y") FROM example;
```
### Describe alternatives you've considered
This can be achieved via unnest and array_agg, but is somewhat painful to do so, requires the existence of a preserved row_id for group_by operation, and introduces uncertainty as to preservation of ordering. It does not appear that datafusion supports `WITH ORDINALITY` which would be used to orderwise guarantee ordering is maintained.
Example:
```
CREATE TABLE example AS
SELECT * FROM (
VALUES
(1, ARRAY[NAMED_STRUCT('x', 1.0, 'y', 2.0),
NAMED_STRUCT('x', 3.0, 'y', 4.0)]),
(2, ARRAY[NAMED_STRUCT('x', 5.0, 'y', 6.0)])
) as example(id, position2d);
select array_agg(p.x) as x, array_agg(p.y) as y
FROM (select id, unnest(position2d) as p from example)
GROUP BY id ORDER BY id;
```
### Additional context
Structurally, the appropriate child array of the struct should be able to be used with the offset array from the list-array and I believe the "right thing" should happen. As such I believe this should generally be able to be implemented as a cheap operation along the lines of a cast.
Contributor guide
Research direction
Start with the SQL examples using array_field, unnest, and array_agg, then trace how list arrays and struct fields are represented and projected. Done means SQL can expose each inner struct field as an array while preserving the list offsets and element ordering, with coverage for the examples shown in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100