apache / apache/datafusion-comet
[EPIC] Implement all Spark date/time expressions
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## What is the problem the feature request solves?
This epic tracks support for the full set of Spark date/time expressions in Comet. Each expression below has a checkbox indicating whether it is implemented today. In-flight work is linked to the relevant PR. Versions in parentheses indicate the Spark version that introduced the expression; unmarked entries are present in all supported Spark versions (3.4 / 3.5 / 4.0 / 4.1).
The canonical list is derived from `FunctionRegistry.scala` in Spark 4.0.1 and Spark 4.1.1 (the `// datetime functions` section), augmented with the `TimeType` expressions defined in `timeExpressions.scala` (4.1+).
## Date / Timestamp functions
- [x] `AddMonths` (`add_months`) - #4417
- [ ] `CurrentDate` (`current_date`, `curdate`) - typically constant-folded by Spark before plan rewriting; only relevant when constant folding is disabled (#3448)
- [ ] `CurrentTimestamp` (`current_timestamp`) - typically constant-folded; see above
- [ ] `Now` (`now`) - typically constant-folded; see above
- [x] `CurrentTimeZone` (`current_timezone`)
- [x] `LocalTimestamp` (`localtimestamp`)
- [x] `DateDiff` (`datediff`, `date_diff`)
- [x] `DateAdd` (`date_add`, `dateadd`)
- [x] `DateFormatClass` (`date_format`)
- [x] `DateSub` (`date_sub`)
- [x] `DayOfMonth` (`day`, `dayofmonth`)
- [x] `DayOfYear` (`dayofyear`)
- [x] `FromUnixTime` (`from_unixtime`)
- [x] `FromUTCTimestamp` (`from_utc_timestamp`)
- [x] `Hour` (`hour`)
- [x] `LastDay` (`last_day`)
- [x] `Minute` (`minute`)
- [x] `Month` (`month`)
- [x] `MonthsBetween` (`months_between`) - #4417
- [x] `NextDay` (`next_day`)
- [x] `Quarter` (`quarter`)
- [x] `Second` (`second`)
- [ ] `ParseToTimestamp` (`to_timestamp`) - partial via #3467
- [ ] `ParseToTimestampNTZ` (`to_timestamp_ntz`)
- [ ] `ParseToTimestampLTZ` (`to_timestamp_ltz`)
- [ ] `ParseToDate` (`to_date`) - #3467
- [x] `ToUnixTimestamp` (`to_unix_timestamp`) - #4417
- [x] `ToUTCTimestamp` (`to_utc_timestamp`)
- [x] `TruncDate` (`trunc`)
- [x] `TruncTimestamp` (`date_trunc`)
- [x] `UnixTimestamp` (`unix_timestamp`)
- [x] `DayOfWeek` (`dayofweek`)
- [x] `WeekDay` (`weekday`)
- [ ] `DayName` (`dayname`) (3.5+)
- [x] `WeekOfYear` (`weekofyear`)
- [x] `Year` (`year`)
- [x] `MakeDate` (`make_date`)
- [x] `MakeTimestamp` (`make_timestamp`) - #4417
- [ ] `MakeTimestampNTZ` (`make_timestamp_ntz`) - #3108
- [ ] `MakeTimestampLTZ` (`make_timestamp_ltz`)
- [ ] `TryMakeTimestamp` (`try_make_timestamp`) - #3117
- [ ] `TryMakeTimestampNTZ` (`try_make_timestamp_ntz`)
- [ ] `TryMakeTimestampLTZ` (`try_make_timestamp_ltz`)
- [ ] `MonthName` (`monthname`) (4.0+)
- [x] `Extract` (`extract(field FROM source)`)
- [x] `DatePart` (`date_part`, `datepart`)
- [x] `DateFromUnixDate` (`date_from_unix_date`)
- [x] `UnixDate` (`unix_date`)
- [x] `SecondsToTimestamp` (`timestamp_seconds`)
- [x] `MillisToTimestamp` (`timestamp_millis`) - #4417
- [x] `MicrosToTimestamp` (`timestamp_micros`) - #4417
- [x] `UnixSeconds` (`unix_seconds`) - #4417
- [x] `UnixMillis` (`unix_millis`) - #4417
- [x] `UnixMicros` (`unix_micros`) - #4417
- [x] `ConvertTimezone` (`convert_timezone`)
## Interval construction
- [ ] `MakeInterval` (`make_interval`) - #3099
- [ ] `MakeDTInterval` (`make_dt_interval`) - #4338
- [ ] `MakeYMInterval` (`make_ym_interval`) - #3100
- [ ] `TryMakeInterval` - #3103
## Timestamp arithmetic (related catalyst expressions)
- [ ] `TimestampAdd` (`timestampadd`) - #3113
- [ ] `TimestampAddInterval` - #3114
- [ ] `TimestampAddYMInterval` - #3115
- [ ] `TimestampDiff` (`timestampdiff`) - #3116
- [ ] `SubtractDates` - #3094
- [ ] `SubtractTimestamps` - #3112
- [ ] `DateTimeSub` - #3134
## Time-window expressions
- [ ] `TimeWindow` (`window`)
- [ ] `SessionWindow` (`session_window`)
- [ ] `WindowTime` (`window_time`) - #3732 (via `PreciseTimestampConversion`)
## TimeType expressions (Spark 4.1+)
See umbrella issue #4288 for the full TimeType implementation plan.
- [x] `ToTime` (`to_time`) - default format only
- [x] `TryToTime` (`try_to_time`) - default format only
- [x] `MakeTime` (`make_time`)
- [ ] `CurrentTime` (`current_time`) - typically constant-folded
- [ ] `HoursOfTime` (`extract(HOUR FROM t)`) - #3126, #3615
- [ ] `MinutesOfTime` (`extract(MINUTE FROM t)`) - #3447
- [ ] `SecondsOfTime` (`extract(SECOND FROM t)`) - #3129, #3618
- [ ] `SecondsOfTimeWithFraction`
- [ ] `TimeFromSeconds` (`time_from_seconds`)
- [ ] `TimeFromMillis` (`time_from_millis`)
- [ ] `TimeFromMicros` (`time_from_micros`)
- [ ] `TimeToSeconds` (`time_to_seconds`)
- [ ] `TimeToMillis` (`time_to_millis`)
- [ ] `TimeToMicros` (`time_to_micros`)
- [ ] `TimeAddInterval`
- [ ] `SubtractTimes`
- [ ] `TimeDiff` (`time_diff`) - #3122
- [ ] `TimeTrunc` (`time_trunc`) - #3123
## Related work
- #4311 - Implement JVM UDFs for all date/time expressions (delivery strategy: route gaps through the codegen dispatcher for Spark-compatible defaults, native implementations remain opt-in)
- #4288 - TimeType support epic (covers infrastructure: shuffle, sort, min/max, casts)
- #3448 - Scalar inputs for datetime expressions when constant folding is disabled (covers `current_date`/`current_timestamp`/`now` regression cases)
- #2649 - `date_trunc` incorrect results in non-UTC timezone
- #3180 - `hour`/`minute`/`second` incorrect timezone handling for `TimestampNTZ`
### Describe the potential solution
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.