apache / apache/datafusion-comet
Hash functions fall back to Spark for decimal precision > 18, and `sha2` for a non-literal `numBits`
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Part of #5572.
`HashUtils.unsupportedReasonFor` declines a `DecimalType` with precision > 18, walking into structs, arrays and map key/value types to find one:
```scala
case d: DecimalType if d.precision > 18 => Some(unsupportedDecimalReason)
```
`spark/src/main/scala/org/apache/comet/serde/hash.scala:137`
Four serdes route through it and none mixes in `CodegenDispatchFallback`: `CometMurmur3Hash` (`hash.scala:49`), `CometXxHash64` (`:27`), `CometSha1` (`:101`) and `CometSha2` (`:75`). So `hash(high_precision_decimal_col)` or `xxhash64(...)` fails the whole projection back to Spark, which matters because these turn up in bucketing, partitioning and dedup paths where the surrounding operator is worth keeping native.
The reason given — Spark hashes via Java `BigDecimal` — is again a case where the dispatcher is the right answer rather than a native fix, since running Spark's `HashExpression.doGenCode` reproduces the `BigDecimal` hashing exactly. `DecimalType` is in `CometBatchKernelCodegen.isSupportedDataType` at any precision.
`CometSha2` has a second decline worth covering in the same change:
```scala
if (!expr.right.foldable) {
Unsupported(Some(nonFoldableNumBitsReason))
}
```
`hash.scala:84-85` — a `numBits` argument that is a column rather than a literal. `Sha2.doGenCode` handles that fine.
Not in scope: the `TimeType` arm of the same helper. `isTimeType` values are admitted by the kernel via `case dt if isTimeType(dt) => true`, so it may be dispatchable too, but Comet's `TimeType` support is new enough that it deserves its own look rather than being folded in here.
Contributor guide
Assessment
This issue has not been assessed yet.