apache / apache/datafusion-comet

Delegate int/float/boolean to decimal cast arms to arrow safe cast

Open
#5,095 3 comments 0 reactions 1 assignee Claimed by @0lai0 View on GitHub
area:expressions enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### What is the problem the feature request solves?

The numeric-to-decimal cast arms in `native/spark-expr/src/conversion_funcs/` re-implement arrow's cast algorithms (verified line-for-line against arrow-cast 58.4.0):

- `numeric.rs` `cast_int_to_decimal128_internal`: per-row `checked_mul(10^scale)` plus precision validation, null on overflow (Legacy/Try) or Spark error (ANSI). Arrow's `cast_integer_to_decimal` with `safe: true` runs the identical algorithm.
- `numeric.rs` `cast_floating_point_to_decimal128`: the fast path (`(f * mul).round()` through f64, filter on precision) matches arrow's `cast_floating_point_to_decimal` safe path exactly, including going through f64 for Float32 and half-away-from-zero rounding. Both share the `10_f64.powi` imprecision tracked in #1371, so results stay bit-identical.
- `boolean.rs` `cast_boolean_to_decimal`: per-row map to `10^scale` or 0. Composable as arrow `cast(Boolean -> Int8)` then `cast(Int8 -> Decimal128(p, s))`.

### Describe the potential solution

Replace the vectorized passes with `cast_with_options(safe: true)` and keep the existing thin Spark wrappers:

- For ANSI, keep the existing rescan-on-error pattern (already used by the float path: compare output null count against input null count, rescan to find the offending value, and raise `SparkError::NumericValueOutOfRange` with the original unscaled value). Arrow's `safe: false` error text is not Spark-shaped, so error remap-by-parsing is not viable.
- Boolean-to-decimal currently errors on overflow in all eval modes, so use `safe: false` plus remap, or keep the existing precision pre-check.

### Additional context

Negative scale is impossible from Spark here, so arrow's negative-scale division branch never fires.

Found during an audit of native code that replicates existing arrow-rs kernels.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.