apache / apache/datafusion

`UnwrapCastInComparison` produces incorrect results

Open
#14,303 7 comments 0 reactions 1 assignee Claimed by @Spaarsh View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

I found that `UnwrapCastInComparison` always assumes the cast operation can succeed, but when it cannot, it results in incorrect optimization results.

### To Reproduce

Run query in CLI (compiled from the latest main: https://github.com/apache/datafusion/commit/f77579108d1dc0285636fbfb24507d2bfca66446)

```sh
DataFusion CLI v44.0.0
> with t as (select 1000000 as a) select try_cast(a as smallint) > 1 from t;
+----------------+
| t.a > Int64(1) |
+----------------+
| true |
+----------------+
1 row(s) fetched.
Elapsed 0.008 seconds.

> with t as (select 1000000 as a) select cast(a as smallint) > 1 from t;
+----------------+
| t.a > Int64(1) |
+----------------+
| true |
+----------------+
1 row(s) fetched.
Elapsed 0.007 seconds.
```

### Expected behavior

When optimizations are disabled, the above queries will produce different results, which are correct.
```sh
> set datafusion.optimizer.max_passes=0;
0 row(s) fetched.
Elapsed 0.003 seconds.

> with t as (select 1000000 as a) select try_cast(a as smallint) > 1 from t;
+----------------+
| t.a > Int64(1) |
+----------------+
| NULL |
+----------------+
1 row(s) fetched.
Elapsed 0.006 seconds.

> with t as (select 1000000 as a) select cast(a as smallint) > 1 from t;
Arrow error: Cast error: Can't cast value 1000000 to type Int16
```

### Additional context

I don't think this is a very urgent bug because both Spark and DuckDB have similar issues.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.