apache / apache/datafusion-comet

unix_timestamp support level disagrees with documented TimestampNTZ tz-conversion divergence

Open
#4,682 1 comment 0 reactions 1 assignee Claimed by @comphead View on GitHub
area:expressions documentation priority:low temporal expressions
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### Describe the bug

`CometUnixTimestamp` (`spark/src/main/scala/org/apache/comet/serde/datetime.scala`) gives inconsistent signals for `TimestampNTZType` input:

- `getUnsupportedReasons` states `TimestampNTZType is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values`.
- `isSupportedInputType` returns `true` for `TimestampNTZType`, and `getSupportLevel` reports `Compatible()` for it.

So the EXPLAIN-time reason and the runtime behavior disagree. If the native path really does apply session-timezone conversion to `TIMESTAMP_NTZ` (as the documented divergence on `unix_timestamp` in `docs/source/contributor-guide/spark_expressions_support.md` suggests), then Comet produces a different value than Spark for non-UTC session timezones. If it does not, the reason text is stale.

This is the `unix_timestamp` / `UnixTimestamp` analog of the hour/minute/second TimestampNTZ tz-conversion bug tracked in #3180.

### Steps to reproduce

With a non-UTC `spark.sql.session.timeZone`, evaluate `unix_timestamp` (and `to_unix_timestamp`) on a `TIMESTAMP_NTZ` column and compare Comet against Spark.

### Expected behavior

The EXPLAIN-time reason must match runtime behavior. If the native path applies session-timezone conversion to `TIMESTAMP_NTZ`, the support level for that input type should be `Incompatible(Some(...))` (cross-referencing #3180); if it does not, the stale reason text should be removed and the inconsistency resolved.

### Additional context

Split out from #4502 (item 1), surfaced by the `audit-comet-expression` skill run in #4448. Cross-references #3180 (same tz-conversion divergence for hour/minute/second on TimestampNTZ).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.