apache / apache/datafusion-comet

`timestamp_seconds` falls back to Spark for decimal, byte and short input

Open
#5,588 1 comment 0 reactions 1 assignee Claimed by @0lai0 View on GitHub
area:expressions enhancement temporal expressions
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Part of #5572.

`timestamp_seconds` accepts only int, long, float and double, and falls the whole projection back to Spark for anything else:

```scala
case IntegerType | LongType | FloatType | DoubleType => Compatible()
case dt => Unsupported(Some(s"timestamp_seconds does not support input type $dt"))
```

`spark/src/main/scala/org/apache/comet/serde/datetime.scala:495-496`

The serde's own `getUnsupportedReasons` spells out what that leaves behind: "`DecimalType`, `ByteType`, and `ShortType` fall back to Spark."

The decimal case is the one that matters. `timestamp_seconds(decimal_col)` is how you get sub-microsecond-safe epoch conversion in Spark, and `SecondsToTimestamp.doGenCode` has a dedicated decimal branch that does the rounding Spark's semantics require. `DecimalType`, `ByteType` and `ShortType` are all in `CometBatchKernelCodegen.isSupportedDataType`.

Fix: mix `CodegenDispatchFallback` into `CometSecondsToTimestamp`. It currently extends `CometScalarFunction[SecondsToTimestamp]`, which is fine — the mixin composes with it, as `CometConcat` and `CometMapFromEntries` already demonstrate.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.