apache / apache/datafusion-comet

Hashing a CalendarInterval value fails with "Unsupported data type in hasher: Interval(MonthDayNano)"

Open
#5,059 3 comments 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:expressions bug priority:high
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### Describe the bug

Hashing a `CalendarIntervalType` value fails the query with a native error instead of either working or falling back to Spark:

```
org.apache.comet.CometNativeException: Unsupported data type in hasher: Interval(MonthDayNano).
This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues
at org.apache.comet.Native.executePlan(Native Method)
at org.apache.comet.CometExecIterator.$anonfun$getNextBatch$2(CometExecIterator.scala:155)
...
```

#4898 added `CalendarIntervalType` to `QueryPlanSerde.supportedDataType`:

https://github.com/apache/datafusion-comet/blob/1d3044f35bda457a20f1ccf4324a4699b35f5158/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala#L530-L534

That is the gate `hash.scala` consults when deciding whether a hash expression's child type is supported:

https://github.com/apache/datafusion-comet/blob/1d3044f35bda457a20f1ccf4324a4699b35f5158/spark/src/main/scala/org/apache/comet/serde/hash.scala#L136-L147

So `Murmur3Hash` / `XxHash64` over a CalendarInterval child now reports `Compatible` and is serialized to native, but `native/spark-expr/src/hash_funcs/utils.rs` has no branch for `Interval(MonthDayNano)` and falls through to the catch-all `DataFusionError::Internal("Unsupported data type in hasher: ...")`.

The ANSI interval types are not in `supportedDataType`, so `hash(dt_interval)` and `hash(ym_interval)` correctly fall back to Spark. Only `CalendarIntervalType` reaches native and fails.

### Steps to reproduce

Against a Parquet-backed table so the expression is not constant-folded away:

```sql
CREATE TABLE t(d int, h int, y int, m int) USING parquet;
INSERT INTO t VALUES (1, 2, 1, 2), (0, 0, 0, 3);

SELECT hash(make_interval(y, m, 0, d, h, 0, 0)) FROM t;
```

Equivalently, as a sql-file test under `spark/src/test/resources/sql-tests/expressions/datetime/`.

A `OneRowRelation` form reproduces it too:

```sql
SELECT hash(make_interval(1, 2, 3, 4, 5, 6, 7.008009));
```

Physical plan at failure time (the whole projection is native):

```
CometNativeColumnarToRow
+- CometProject [hash(make_interval(1, 2, 3, 4, 5, 6, cast(7.008009 as decimal(18,6)), false), 42) ...]
+- CometSparkRowToColumnar
+- *(1) Scan OneRowRelation[]
```

### Expected behavior

The query should return the same values as Spark. Failing that, it should fall back to Spark rather than aborting the stage.

Two possible fixes:

1. Implement Spark-compatible hashing for `Interval(MonthDayNano)` (and `Interval(YearMonth)` / `Duration(Microsecond)` while we are there) in `native/spark-expr/src/hash_funcs/utils.rs`. Spark's `HashExpression` hashes a `CalendarInterval` as its `months`, `days` and `microseconds` fields, so the native side needs to match that field order and width exactly.
2. Or narrow the Scala gate so `CalendarIntervalType` is rejected for hash expressions, matching how the ANSI interval types behave today, and leave native hashing for the follow-up that implements it properly.

Either way this needs a regression test covering `hash` and `xxhash64` over all three interval types.

### Additional context

- Found while auditing interval support after #4898 and #4976 landed. Part of #4540.
- Reproduced on `1d3044f35` with the Spark 4.1 profile.
- `QueryPlanSerde.supportedDataType` is also consulted by `CometScalarSubquery` and `CometSink`, so it is worth checking whether a CalendarInterval-typed scalar subquery hits a similar unhandled-type path in the planner.
- Related gap worth noting separately: native shuffle accepts all three interval types after #4976, but `supportedHashPartitioningDataType` in `CometShuffleExchangeExec` rejects them, so hash partitioning on an interval key still falls back. That is a fallback rather than a failure, so it is not part of this issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.