apache / apache/datafusion

CAST statistics can produce incorrect MIN/MAX results

Open
#25,221 1 comment 0 reactions 1 assignee Claimed by @haohuaijin View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Casting string min/max statistics to integers does not preserve their ordering. Aggregate optimization can therefore return incorrect results.

### To Reproduce

```sql
SET datafusion.execution.target_partitions = 1;
SET datafusion.execution.collect_statistics = true;

COPY (SELECT * FROM (VALUES ('1'), ('100'), ('2')) AS t(a))
TO '/tmp/cast_statistics.parquet' STORED AS PARQUET;

CREATE EXTERNAL TABLE t STORED AS PARQUET
LOCATION '/tmp/cast_statistics.parquet';

SELECT MIN(CAST(a AS INT)), MAX(CAST(a AS INT)) FROM t;
```

Actual: `1, 2`. Expected: `1, 100`. The same issue occurs with BIGINT.

### Expected behavior

Only propagate CAST statistics when the conversion is proven safe for the supplied bounds.

### Additional context

found this when work on https://github.com/apache/datafusion/issues/25220

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.