Spark `round()` on floating-point types diverges from Spark HALF_UP semantics
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
The Spark-compatible `round()` function gives different results from Apache Spark when the input is a floating-point type (`FloatType`/`DoubleType`) and the value's binary representation is slightly off from its decimal literal.
Spark's `RoundBase` rounds a double as `BigDecimal(d).setScale(scale, HALF_UP)`, where `BigDecimal(Double)` is `java.math.BigDecimal.valueOf(d)` — i.e. it parses the *shortest round-trip decimal string* of the double (`Double.toString`). DataFusion's `round_float` instead does naive binary-float arithmetic, `(value * 10^scale).round() / 10^scale`, which rounds the already-imprecise binary value and diverges at the half-way point.
### To Reproduce
```sql
SELECT round(1.255::double, 2::int);
-- Spark: 1.26
-- DataFusion: 1.25
SELECT round(1.005::double, 2::int);
-- Spark: 1.01
-- DataFusion: 1.0
```
The cause is that `1.255` and `1.005` are stored as binary doubles a hair below the decimal value (`1.2549999999999999...`, `1.00499999999999989...`). Spark sees the shortest decimal string (`"1.255"`, `"1.005"`) and applies HALF_UP, so the tie rounds away from zero. DataFusion multiplies the raw binary value by `100`, which stays below the half-way point, and rounds down.
### Expected behavior
Match Spark: round via the shortest round-trip decimal representation with HALF_UP (ties away from zero), for both `DoubleType` and `FloatType` (Spark widens float to double first via `f.toDouble`).
### Additional context
The existing doc comment on `round_float` already describes the intended `BigDecimal` / HALF_UP behaviour; the implementation simply doesn't match it. I have a fix and will open a PR referencing this issue.
`datafusion/spark/src/function/math/round.rs`
Contributor guide
Research direction
Start in datafusion/spark/src/function/math/round.rs, especially the round_float entry point and its existing doc comment describing BigDecimal and HALF_UP behavior. Reproduce the 1.255 and 1.005 SQL examples, then add regression coverage showing Spark-compatible results for floating-point inputs, including both FloatType and DoubleType.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, spark, sql
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100