apache / apache/datafusion-comet
Delegate int/float/boolean to decimal cast arms to arrow safe cast
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### What is the problem the feature request solves?
The numeric-to-decimal cast arms in `native/spark-expr/src/conversion_funcs/` re-implement arrow's cast algorithms (verified line-for-line against arrow-cast 58.4.0):
- `numeric.rs` `cast_int_to_decimal128_internal`: per-row `checked_mul(10^scale)` plus precision validation, null on overflow (Legacy/Try) or Spark error (ANSI). Arrow's `cast_integer_to_decimal` with `safe: true` runs the identical algorithm.
- `numeric.rs` `cast_floating_point_to_decimal128`: the fast path (`(f * mul).round()` through f64, filter on precision) matches arrow's `cast_floating_point_to_decimal` safe path exactly, including going through f64 for Float32 and half-away-from-zero rounding. Both share the `10_f64.powi` imprecision tracked in #1371, so results stay bit-identical.
- `boolean.rs` `cast_boolean_to_decimal`: per-row map to `10^scale` or 0. Composable as arrow `cast(Boolean -> Int8)` then `cast(Int8 -> Decimal128(p, s))`.
### Describe the potential solution
Replace the vectorized passes with `cast_with_options(safe: true)` and keep the existing thin Spark wrappers:
- For ANSI, keep the existing rescan-on-error pattern (already used by the float path: compare output null count against input null count, rescan to find the offending value, and raise `SparkError::NumericValueOutOfRange` with the original unscaled value). Arrow's `safe: false` error text is not Spark-shaped, so error remap-by-parsing is not viable.
- Boolean-to-decimal currently errors on overflow in all eval modes, so use `safe: false` plus remap, or keep the existing precision pre-check.
### Additional context
Negative scale is impossible from Spark here, so arrow's negative-scale division branch never fires.
Found during an audit of native code that replicates existing arrow-rs kernels.
Contributor guide
Assessment
This issue has not been assessed yet.