apache / apache/datafusion-comet

`unix_timestamp` on string input falls back to Spark, while `to_unix_timestamp` already uses the codegen dispatcher

Open
#5,577 1 comment 0 reactions 1 assignee Claimed by @Satyr09 View on GitHub
area:expressions enhancement performance temporal expressions
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Part of #5572.

`CometUnixTimestamp` only accepts date and timestamp input; anything else, which in practice means **string**, falls the whole projection back to Spark:

```scala
Unsupported(Some(s"unix_timestamp does not support input type: $inputType"))
```

`spark/src/main/scala/org/apache/comet/serde/datetime.scala:319`

`unix_timestamp(str, fmt)` on a string column is the common spelling of this function, so the supported case is arguably the rarer one. `UnixTimestamp` is a normal codegen expression, and `StringType` is in `CometBatchKernelCodegen.isSupportedDataType`, so the dispatcher handles it.

`to_unix_timestamp` is already Hybrid — `CometToUnixTimestamp` is a `CometCodegenDispatch[ToUnixTimestamp]` (`serde/datetime.scala:929`) — so the two functions, which differ only in argument order, behave completely differently today.

The same serde also returns a bare `Incompatible(collationReason)` for collated input (`:314`), which the same mixin would route through the dispatcher rather than falling back.

Two things to watch:

- The input-type check is duplicated in `convert` (`:329`). Only the `getSupportLevel` copy is reachable by the dispatcher, so the `convert` copy should be removed as part of this change or it will mask the fix. See the prerequisite issue on `convert`-side declines.
- Parser-policy behavior. `spark.sql.legacy.timeParserPolicy` is honored by Spark's generated code, so routing through the dispatcher preserves whatever policy the session selected — see the note in `docs/source/contributor-guide/spark_configs_support.md`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.