apache / apache/datafusion

Inconsistent and incorrect `struct` field coercion

Open
#14,396 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

When coercing structs with different types DataFusion is inconsistent in its behavior. Sometimes it errors and in other times it is inconsistent

### To Reproduce

Consider two structs with two fields
```sql
create table t as values
(
{ 'foo': 'baz' },
{ 'xxx': arrow_cast('blarg', 'Utf8View') } -- column4 has List of Struct w/ Utf8View and a different field name
);
```

They can be coerced to the same field using `UNION`,

```
> select column1 from t UNION ALL select column2 from t;
+-------------+
| column1 |
+-------------+
| {c0: baz} |
| {c0: blarg} |
+-------------+
2 row(s) fetched.
Elapsed 0.018 seconds.
```

Likewise with `CASE` they are coerced to the same value
```sql
> select CASE WHEN 1=2 THEN column1 ELSE column2 END from t ;
+-----------------------------------------------------------------+
| CASE WHEN Int64(1) = Int64(2) THEN t.column1 ELSE t.column2 END |
+-----------------------------------------------------------------+
| {c0: blarg} |
+-----------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.010 seconds.
```

### Expected behavior

I believe the field names will be different after
- https://github.com/apache/datafusion/issues/14383
- https://github.com/apache/datafusion/issues/14154

However I expect all of the above queries to fail as the field names are incorrect

I expect the following cases to work:
1. Field names are the same but in different order
2. Field names are the same but the value types need to be coerced themselves (see case on https://github.com/apache/datafusion/issues/14154)

### Additional context

@jayzhan211 suggests: https://github.com/apache/datafusion/pull/14384/files#r1937492704

> Yes, I think type union resolution is the correct on for CASE

Contributor guide

Open the contributing guide

Research direction

Reproduce the UNION and CASE SQL examples from the issue, then trace DataFusion's struct coercion and type-union resolution behavior. Compare the results with the expected cases: reject incompatible field names, while accepting matching names in a different order or with coercible value types; consider the related issues #14383 and #14154.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.