apache / apache/datafusion

Sub-field names are not handled correctly when combining named_structs and NULL structs

Open
#14,632 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

When a Substrait plan contains e.g. an CASE WHEN statement, where one arm returns a named struct, and another arm returns a non-named struct, the names are handled incorrectly.

### To Reproduce

Add the following to roundtrip_logical_plans.rs:
```
#[tokio::test]
async fn roundtrip_named_struct() -> Result<()> {
roundtrip("SELECT (CASE WHEN column1 IS NOT NULL THEN named_struct('f1', column1) ELSE NULL END) as res FROM (VALUES ('a'), (NULL))").await
}
```

This results in an error:
```
assertion `left == right` failed
left: "Projection: CASE WHEN column1 IS NOT NULL THEN named_struct(Utf8(\"f1\"), column1) ELSE Struct({f1:}) END AS res\n Values: (Utf8(\"a\")), (Utf8(NULL))"
right: "Projection: CASE WHEN column1 IS NOT NULL THEN CAST(named_struct(Utf8(\"f1\"), column1) AS Struct([Field { name: \"c0\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }])) ELSE Struct({c0:}) END AS res\n Values: (Utf8(\"a\")), (Utf8(NULL))"
```

The test might fail anyways, but what's meaningful is that the error says DF is trying to cast the named_struct into having "c0" as the field name, rather than "f1" as was intended. This is despite the Substrait plan defining the output names correctly: `names: ["res", "f1"]`. The issue is likely due to the two arms of the case-when statement having differing output types, which confuses the renaming logic somehow.

### Expected behavior

The names of the named_struct should take precedence, if names mismatch then an error should likely be thrown.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by adding the reported roundtrip_named_struct test to roundtrip_logical_plans.rs and run it to reproduce the assertion. Then trace the CASE expression's differing struct outputs and named-field handling. Done means the round-trip preserves the intended f1 field name, or reports a clear mismatch error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.