Align procedure decimal filters with Spark 4.1.1 SQL semantics
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Current status
Updated for #19836 at `df61fa197caa`. The originally suspected extreme decimal/decimal mismatch has now been checked at runtime and matches Spark 4.1.1. Two additional mixed decimal/integer comparison mismatches found during review have been fixed by invoking Spark's decimal comparison rules before generic widening.
| Coercion case | Current behavior | Spark parity |
| --- | --- | --- |
| Integral/floating comparisons, either order | Widens using the active ANSI rules | Matches tested cases on 3.5.5 and 4.1.1 |
| Numeric `IN`, `<=>`, and `coalesce`, including nulls | Converts to a common numeric type | Matches tested cases on both |
| Integral arithmetic and division | Widens operands; non-decimal `/` uses Double | Matches tested cases on both |
| Decimal arithmetic with decimal/integral/floating/null operands | Uses Spark's decimal promotion rules without unnecessary common-type casts | Matches tested cases on both |
| Arithmetic overflow and division by zero | Propagates arithmetic exceptions; retains Spark's non-ANSI behavior | Matches tested cases on both |
| `DECIMAL(38,0)` vs `DECIMAL(38,18)` | Uses the active version's decimal conversion | Matches tested cases, including both Spark 4.1.1 legacy decimal settings |
| High-precision decimal/integer comparisons | Applies Spark's decimal comparison rules before generic widening | Both reported mismatches fixed; matches tested cases on both |
### Verified Spark 4.1.1 decimal example
For `big DECIMAL(38,0) = 10^30`, `frac DECIMAL(38,18) = 1.5`, and `big > frac`, Hudi validation succeeds and the tested outcomes match SQL:
| `legacy.decimal.retainFractionDigitsOnTruncate` | ANSI | Common type | Outcome |
| --- | --- | --- | --- |
| false | false | DECIMAL(38,0) | Keeps row |
| false | true | DECIMAL(38,0) | Keeps row |
| true | false | DECIMAL(38,18) | Null from overflow; excludes row |
| true | true | DECIMAL(38,18) | Throws arithmetic overflow |
Both configuration names have the `spark.sql.` prefix; ANSI is `spark.sql.ansi.enabled`. The legacy decimal setting defaults to false. Rounding-sensitive `big = frac` with values `2` and `1.5` was also checked: it matches after rounding under the default setting and does not match under the legacy setting.
These are runtime observations, superseding the earlier source-inspection-only status. They do not establish parity for every decimal expression or configuration.
### Remaining verification and coverage
- Complete the full Spark 4.1 utility/procedure suite rerun with consistent Scala 2.13 Hudi dependencies. Standalone Spark 4.1.1 SQL-parity checks already pass.
- Add the extreme decimal/decimal and rounding-sensitive equality cases above to the committed Spark 4 test matrix. The existing extreme-pair unit test still skips Spark 4; current evidence for that pair is from standalone checks.
- Broaden decimal arithmetic configuration coverage as needed, including `spark.sql.decimalOperations.allowPrecisionLoss=false`; the committed mixed-arithmetic parity matrix explicitly tests true.
Current verification: 28 Spark 3.5 utility tests passed; 360 SQL-parity assertions passed per version on Spark 3.5.5 and 4.1.1; scalastyle clean. No blanket Spark SQL parity claim.
Reference: [Spark 4.1.1 DecimalPrecisionTypeCoercion](https://github.com/apache/spark/blob/v4.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionTypeCoercion.scala).
### Related Issues
- #19836
- #19632
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with Spark's DecimalPrecisionTypeCoercion reference and the existing extreme-pair unit test, which currently skips Spark 4. Add the extreme decimal/decimal and rounding-sensitive equality cases to the committed Spark 4 test matrix, then rerun the full Spark 4.1 utility/procedure suite with consistent Scala 2.13 Hudi dependencies. Done means the added cases and broader configuration coverage match the observed Spark 4.1.1 behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100