apache / apache/datafusion-comet
Include days in CalendarInterval hash when Spark 4.3+ is supported (SPARK-58236)
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Spark <= 4.2 codegen omits the `days` field when hashing `CalendarIntervalType` (`hash` / `xxhash64`), and Comet's native hasher matches that (#5135). [SPARK-58236](https://issues.apache.org/jira/browse/SPARK-58236) (apache/spark commit `a2616be0f`) fixes Spark's codegen to include `days` starting in Spark 4.3, aligning it with the interpreted path.
When Comet adds a Spark 4.3 profile, the hash becomes version-dependent:
- resolve an `include_days` boolean in `ShimCometConf` per the shim convention
- carry it through the protobuf hash expression
- branch on it in `hash_array_interval_month_day_nano!` in `native/spark-expr/src/hash_funcs/utils.rs`
Until then, all supported versions (`v3.4.3`, `v3.5.8`, `v4.0.1`, `v4.1.2`, `v4.2.0`) omit `days` and the fix has not been backported to any release branch, so no native change is needed yet.
Follow-up from review discussion on #5135.
Contributor guide
Assessment
This issue has not been assessed yet.