apache / apache/datafusion-comet
`unix_timestamp` on string input falls back to Spark, while `to_unix_timestamp` already uses the codegen dispatcher
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Part of #5572.
`CometUnixTimestamp` only accepts date and timestamp input; anything else, which in practice means **string**, falls the whole projection back to Spark:
```scala
Unsupported(Some(s"unix_timestamp does not support input type: $inputType"))
```
`spark/src/main/scala/org/apache/comet/serde/datetime.scala:319`
`unix_timestamp(str, fmt)` on a string column is the common spelling of this function, so the supported case is arguably the rarer one. `UnixTimestamp` is a normal codegen expression, and `StringType` is in `CometBatchKernelCodegen.isSupportedDataType`, so the dispatcher handles it.
`to_unix_timestamp` is already Hybrid — `CometToUnixTimestamp` is a `CometCodegenDispatch[ToUnixTimestamp]` (`serde/datetime.scala:929`) — so the two functions, which differ only in argument order, behave completely differently today.
The same serde also returns a bare `Incompatible(collationReason)` for collated input (`:314`), which the same mixin would route through the dispatcher rather than falling back.
Two things to watch:
- The input-type check is duplicated in `convert` (`:329`). Only the `getSupportLevel` copy is reachable by the dispatcher, so the `convert` copy should be removed as part of this change or it will mask the fix. See the prerequisite issue on `convert`-side declines.
- Parser-policy behavior. `spark.sql.legacy.timeParserPolicy` is honored by Spark's generated code, so routing through the dispatcher preserves whatever policy the session selected — see the note in `docs/source/contributor-guide/spark_configs_support.md`.
Contributor guide
Assessment
This issue has not been assessed yet.