Inconsistent and incorrect `struct` field coercion
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
When coercing structs with different types DataFusion is inconsistent in its behavior. Sometimes it errors and in other times it is inconsistent
### To Reproduce
Consider two structs with two fields
```sql
create table t as values
(
{ 'foo': 'baz' },
{ 'xxx': arrow_cast('blarg', 'Utf8View') } -- column4 has List of Struct w/ Utf8View and a different field name
);
```
They can be coerced to the same field using `UNION`,
```
> select column1 from t UNION ALL select column2 from t;
+-------------+
| column1 |
+-------------+
| {c0: baz} |
| {c0: blarg} |
+-------------+
2 row(s) fetched.
Elapsed 0.018 seconds.
```
Likewise with `CASE` they are coerced to the same value
```sql
> select CASE WHEN 1=2 THEN column1 ELSE column2 END from t ;
+-----------------------------------------------------------------+
| CASE WHEN Int64(1) = Int64(2) THEN t.column1 ELSE t.column2 END |
+-----------------------------------------------------------------+
| {c0: blarg} |
+-----------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.010 seconds.
```
### Expected behavior
I believe the field names will be different after
- https://github.com/apache/datafusion/issues/14383
- https://github.com/apache/datafusion/issues/14154
However I expect all of the above queries to fail as the field names are incorrect
I expect the following cases to work:
1. Field names are the same but in different order
2. Field names are the same but the value types need to be coerced themselves (see case on https://github.com/apache/datafusion/issues/14154)
### Additional context
@jayzhan211 suggests: https://github.com/apache/datafusion/pull/14384/files#r1937492704
> Yes, I think type union resolution is the correct on for CASE
Contributor guide
Research direction
Reproduce the UNION and CASE SQL examples from the issue, then trace DataFusion's struct coercion and type-union resolution behavior. Compare the results with the expected cases: reject incompatible field names, while accepting matching names in a different order or with coercible value types; consider the related issues #14383 and #14154.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100