apache / apache/datafusion-comet
`timestamp_seconds` falls back to Spark for decimal, byte and short input
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Part of #5572.
`timestamp_seconds` accepts only int, long, float and double, and falls the whole projection back to Spark for anything else:
```scala
case IntegerType | LongType | FloatType | DoubleType => Compatible()
case dt => Unsupported(Some(s"timestamp_seconds does not support input type $dt"))
```
`spark/src/main/scala/org/apache/comet/serde/datetime.scala:495-496`
The serde's own `getUnsupportedReasons` spells out what that leaves behind: "`DecimalType`, `ByteType`, and `ShortType` fall back to Spark."
The decimal case is the one that matters. `timestamp_seconds(decimal_col)` is how you get sub-microsecond-safe epoch conversion in Spark, and `SecondsToTimestamp.doGenCode` has a dedicated decimal branch that does the rounding Spark's semantics require. `DecimalType`, `ByteType` and `ShortType` are all in `CometBatchKernelCodegen.isSupportedDataType`.
Fix: mix `CodegenDispatchFallback` into `CometSecondsToTimestamp`. It currently extends `CometScalarFunction[SecondsToTimestamp]`, which is fine — the mixin composes with it, as `CometConcat` and `CometMapFromEntries` already demonstrate.
Contributor guide
Assessment
This issue has not been assessed yet.