apache / apache/hudi

Data skipping treats order-breaking casts as order-preserving (isCastPreservingOrdering)

Open
#19,445 0 comments 0 reactions 1 assignee Claimed by @voonhous View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Describe the problem

`HoodieSparkTypeUtils.isCastPreservingOrdering` (hudi-spark-common) only rejects `String <-> Numeric` pairs and returns `true` for every other cast, including narrowing numeric casts. `BaseHoodieCatalystExpressionUtils.OrderPreservingTransformation` uses it to decide whether a `Cast` over a column can be mapped back to the source attribute for data skipping, and `DataSkippingUtils.translateIntoColumnStatsIndexFilterExpr` then rewrites the predicate over the column's min/max through the cast. A non-monotonic cast makes `[cast(min), cast(max)]` an invalid bound for the cast values, so files containing matching rows can be pruned.

### Repro sketch

bigint column `a`, one file with values `{1, 2147483647, 4294967297}` -> col stats `min=1`, `max=4294967297`. Query filter `cast(a as int) > 100` in non-ANSI mode:

- the matcher accepts the cast, so the filter is translated to `cast(a_maxValue as int) > 100`
- `cast(4294967297L as int)` wraps to `1`, the translated filter is false, the file is pruned
- but the file holds `a=2147483647` whose cast is `2147483647 > 100` -> silently missing row

### Related gaps in the same whitelist

- The `Multiply`/`Divide` arms of `OrderPreservingTransformation` match any literal operand, including negative literals, which reverse ordering.
- On Spark 4.x `StringType` is collation-parameterized; casts to/from a non-default-collation string do not equal the `StringType` companion in the match and fall through to `case _ => true`.

### Suggested fix

For numeric-to-numeric pairs require `Cast.canUpCast(from, to)`; keep the `String <-> Numeric` rejections; handle the collation cases explicitly.

Found while reviewing #19405, which pins the current behavior with a TODO referencing this issue.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.