Data skipping treats order-breaking casts as order-preserving (isCastPreservingOrdering)
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Describe the problem
`HoodieSparkTypeUtils.isCastPreservingOrdering` (hudi-spark-common) only rejects `String <-> Numeric` pairs and returns `true` for every other cast, including narrowing numeric casts. `BaseHoodieCatalystExpressionUtils.OrderPreservingTransformation` uses it to decide whether a `Cast` over a column can be mapped back to the source attribute for data skipping, and `DataSkippingUtils.translateIntoColumnStatsIndexFilterExpr` then rewrites the predicate over the column's min/max through the cast. A non-monotonic cast makes `[cast(min), cast(max)]` an invalid bound for the cast values, so files containing matching rows can be pruned.
### Repro sketch
bigint column `a`, one file with values `{1, 2147483647, 4294967297}` -> col stats `min=1`, `max=4294967297`. Query filter `cast(a as int) > 100` in non-ANSI mode:
- the matcher accepts the cast, so the filter is translated to `cast(a_maxValue as int) > 100`
- `cast(4294967297L as int)` wraps to `1`, the translated filter is false, the file is pruned
- but the file holds `a=2147483647` whose cast is `2147483647 > 100` -> silently missing row
### Related gaps in the same whitelist
- The `Multiply`/`Divide` arms of `OrderPreservingTransformation` match any literal operand, including negative literals, which reverse ordering.
- On Spark 4.x `StringType` is collation-parameterized; casts to/from a non-default-collation string do not equal the `StringType` companion in the match and fall through to `case _ => true`.
### Suggested fix
For numeric-to-numeric pairs require `Cast.canUpCast(from, to)`; keep the `String <-> Numeric` rejections; handle the collation cases explicitly.
Found while reviewing #19405, which pins the current behavior with a TODO referencing this issue.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.