apache / apache/datafusion-comet
Hashing a CalendarInterval value fails with "Unsupported data type in hasher: Interval(MonthDayNano)"
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### Describe the bug
Hashing a `CalendarIntervalType` value fails the query with a native error instead of either working or falling back to Spark:
```
org.apache.comet.CometNativeException: Unsupported data type in hasher: Interval(MonthDayNano).
This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues
at org.apache.comet.Native.executePlan(Native Method)
at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:155)
...
```
#4898 added `CalendarIntervalType` to `QueryPlanSerde.supportedDataType`:
https://github.com/apache/datafusion-comet/blob/1d3044f35bda457a20f1ccf4324a4699b35f5158/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala#L530-L534
That is the gate `hash.scala` consults when deciding whether a hash expression's child type is supported:
https://github.com/apache/datafusion-comet/blob/1d3044f35bda457a20f1ccf4324a4699b35f5158/spark/src/main/scala/org/apache/comet/serde/hash.scala#L136-L147
So `Murmur3Hash` / `XxHash64` over a CalendarInterval child now reports `Compatible` and is serialized to native, but `native/spark-expr/src/hash_funcs/utils.rs` has no branch for `Interval(MonthDayNano)` and falls through to the catch-all `DataFusionError::Internal("Unsupported data type in hasher: ...")`.
The ANSI interval types are not in `supportedDataType`, so `hash(dt_interval)` and `hash(ym_interval)` correctly fall back to Spark. Only `CalendarIntervalType` reaches native and fails.
### Steps to reproduce
Against a Parquet-backed table so the expression is not constant-folded away:
```sql
CREATE TABLE t(d int, h int, y int, m int) USING parquet;
INSERT INTO t VALUES (1, 2, 1, 2), (0, 0, 0, 3);
SELECT hash(make_interval(y, m, 0, d, h, 0, 0)) FROM t;
```
Equivalently, as a sql-file test under `spark/src/test/resources/sql-tests/expressions/datetime/`.
A `OneRowRelation` form reproduces it too:
```sql
SELECT hash(make_interval(1, 2, 3, 4, 5, 6, 7.008009));
```
Physical plan at failure time (the whole projection is native):
```
CometNativeColumnarToRow
+- CometProject [hash(make_interval(1, 2, 3, 4, 5, 6, cast(7.008009 as decimal(18,6)), false), 42) ...]
+- CometSparkRowToColumnar
+- *(1) Scan OneRowRelation[]
```
### Expected behavior
The query should return the same values as Spark. Failing that, it should fall back to Spark rather than aborting the stage.
Two possible fixes:
1. Implement Spark-compatible hashing for `Interval(MonthDayNano)` (and `Interval(YearMonth)` / `Duration(Microsecond)` while we are there) in `native/spark-expr/src/hash_funcs/utils.rs`. Spark's `HashExpression` hashes a `CalendarInterval` as its `months`, `days` and `microseconds` fields, so the native side needs to match that field order and width exactly.
2. Or narrow the Scala gate so `CalendarIntervalType` is rejected for hash expressions, matching how the ANSI interval types behave today, and leave native hashing for the follow-up that implements it properly.
Either way this needs a regression test covering `hash` and `xxhash64` over all three interval types.
### Additional context
- Found while auditing interval support after #4898 and #4976 landed. Part of #4540.
- Reproduced on `1d3044f35` with the Spark 4.1 profile.
- `QueryPlanSerde.supportedDataType` is also consulted by `CometScalarSubquery` and `CometSink`, so it is worth checking whether a CalendarInterval-typed scalar subquery hits a similar unhandled-type path in the planner.
- Related gap worth noting separately: native shuffle accepts all three interval types after #4976, but `supportedHashPartitioningDataType` in `CometShuffleExchangeExec` rejects them, so hash partitioning on an interval key still falls back. That is a fallback rather than a failure, so it is not part of this issue.
Contributor guide
Assessment
This issue has not been assessed yet.