apache / apache/datafusion

Sort key over get_field(named_struct(...), f) is evaluated against the materialized struct instead of the base column

Open
#22,487 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Follow-up from #22239. For a query that sorts by a struct field, where the table is `WITH ORDER (a + b)` so a `SortExec` is genuinely required:

```sql
EXPLAIN SELECT named_struct('a', a, 'b', b) AS s FROM ordered ORDER BY s['a'];
```

**Plan before #22239** — `get_field` sort key extracted into a flat scan column, recovery projection hides it:

```
01)ProjectionExec: expr=[s@0 as s]
02)--SortExec: expr=[__datafusion_extracted_1@1 ASC NULLS LAST], preserve_partitioning=[false]
03)----DataSourceExec: projection=[named_struct(a, a@0, b, b@1) as s, get_field(named_struct(a, a@0, b, b@1), a) as __datafusion_extracted_1], ...
```

**Plan after #22239** — simpler, but the sort key is a struct-field extraction over the materialized struct:

```
01)SortExec: expr=[get_field(s@0, a) ASC NULLS LAST], preserve_partitioning=[false]
02)--DataSourceExec: projection=[named_struct(a, a@0, b, b@1) as s], ...
```

**Ideal plan** — sort on the base column, build the struct afterward:

```
01)ProjectionExec: expr=[named_struct(a, a@0, b, b@1) as s]
02)--SortExec: expr=[a@0 ASC NULLS LAST]
03)----DataSourceExec: projection=[a, b]
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the provided EXPLAIN query against the ordered table and compare the current plan with the ideal plan. Trace how the sort key for get_field(named_struct(...), 'a') is extracted and when the struct is materialized; done means SortExec orders by the base column and the projection builds the struct afterward.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.