apache / apache/datafusion-comet

Hash functions fall back to Spark for decimal precision > 18, and `sha2` for a non-literal `numBits`

Open
#5,581 1 comment 0 reactions 1 assignee Claimed by @sam-1112 View on GitHub
area:expressions enhancement performance
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Part of #5572.

`HashUtils.unsupportedReasonFor` declines a `DecimalType` with precision > 18, walking into structs, arrays and map key/value types to find one:

```scala
case d: DecimalType if d.precision > 18 => Some(unsupportedDecimalReason)
```

`spark/src/main/scala/org/apache/comet/serde/hash.scala:137`

Four serdes route through it and none mixes in `CodegenDispatchFallback`: `CometMurmur3Hash` (`hash.scala:49`), `CometXxHash64` (`:27`), `CometSha1` (`:101`) and `CometSha2` (`:75`). So `hash(high_precision_decimal_col)` or `xxhash64(...)` fails the whole projection back to Spark, which matters because these turn up in bucketing, partitioning and dedup paths where the surrounding operator is worth keeping native.

The reason given — Spark hashes via Java `BigDecimal` — is again a case where the dispatcher is the right answer rather than a native fix, since running Spark's `HashExpression.doGenCode` reproduces the `BigDecimal` hashing exactly. `DecimalType` is in `CometBatchKernelCodegen.isSupportedDataType` at any precision.

`CometSha2` has a second decline worth covering in the same change:

```scala
if (!expr.right.foldable) {
Unsupported(Some(nonFoldableNumBitsReason))
}
```

`hash.scala:84-85` — a `numBits` argument that is a column rather than a literal. `Sha2.doGenCode` handles that fine.

Not in scope: the `TimeType` arm of the same helper. `isTimeType` values are admitted by the kernel via `case dt if isTimeType(dt) => true`, so it may be dispatchable too, but Comet's `TimeType` support is new enough that it deserves its own look rather than being folded in here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.