apache / apache/datafusion-comet
unix_timestamp support level disagrees with documented TimestampNTZ tz-conversion divergence
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### Describe the bug
`CometUnixTimestamp` (`spark/src/main/scala/org/apache/comet/serde/datetime.scala`) gives inconsistent signals for `TimestampNTZType` input:
- `getUnsupportedReasons` states `TimestampNTZType is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values`.
- `isSupportedInputType` returns `true` for `TimestampNTZType`, and `getSupportLevel` reports `Compatible()` for it.
So the EXPLAIN-time reason and the runtime behavior disagree. If the native path really does apply session-timezone conversion to `TIMESTAMP_NTZ` (as the documented divergence on `unix_timestamp` in `docs/source/contributor-guide/spark_expressions_support.md` suggests), then Comet produces a different value than Spark for non-UTC session timezones. If it does not, the reason text is stale.
This is the `unix_timestamp` / `UnixTimestamp` analog of the hour/minute/second TimestampNTZ tz-conversion bug tracked in #3180.
### Steps to reproduce
With a non-UTC `spark.sql.session.timeZone`, evaluate `unix_timestamp` (and `to_unix_timestamp`) on a `TIMESTAMP_NTZ` column and compare Comet against Spark.
### Expected behavior
The EXPLAIN-time reason must match runtime behavior. If the native path applies session-timezone conversion to `TIMESTAMP_NTZ`, the support level for that input type should be `Incompatible(Some(...))` (cross-referencing #3180); if it does not, the stale reason text should be removed and the inconsistency resolved.
### Additional context
Split out from #4502 (item 1), surfaced by the `audit-comet-expression` skill run in #4448. Cross-references #3180 (same tz-conversion divergence for hour/minute/second on TimestampNTZ).
Contributor guide
Assessment
This issue has not been assessed yet.