apache / apache/datafusion-comet

Include days in CalendarInterval hash when Spark 4.3+ is supported (SPARK-58236)

Open
#5,498 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:expressions enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Spark <= 4.2 codegen omits the `days` field when hashing `CalendarIntervalType` (`hash` / `xxhash64`), and Comet's native hasher matches that (#5135). [SPARK-58236](https://issues.apache.org/jira/browse/SPARK-58236) (apache/spark commit `a2616be0f`) fixes Spark's codegen to include `days` starting in Spark 4.3, aligning it with the interpreted path.

When Comet adds a Spark 4.3 profile, the hash becomes version-dependent:

- resolve an `include_days` boolean in `ShimCometConf` per the shim convention
- carry it through the protobuf hash expression
- branch on it in `hash_array_interval_month_day_nano!` in `native/spark-expr/src/hash_funcs/utils.rs`

Until then, all supported versions (`v3.4.3`, `v3.5.8`, `v4.0.1`, `v4.1.2`, `v4.2.0`) omit `days` and the fix has not been backported to any release branch, so no native change is needed yet.

Follow-up from review discussion on #5135.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.