apache / apache/datafusion-comet

[EPIC] Implement all Spark date/time expressions

Open
#4,418 0 comments 2 reactions 0 assignees View on GitHub
area:expressions priority:medium
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## What is the problem the feature request solves?

This epic tracks support for the full set of Spark date/time expressions in Comet. Each expression below has a checkbox indicating whether it is implemented today. In-flight work is linked to the relevant PR. Versions in parentheses indicate the Spark version that introduced the expression; unmarked entries are present in all supported Spark versions (3.4 / 3.5 / 4.0 / 4.1).

The canonical list is derived from `FunctionRegistry.scala` in Spark 4.0.1 and Spark 4.1.1 (the `// datetime functions` section), augmented with the `TimeType` expressions defined in `timeExpressions.scala` (4.1+).

## Date / Timestamp functions

- [x] `AddMonths` (`add_months`) - #4417
- [ ] `CurrentDate` (`current_date`, `curdate`) - typically constant-folded by Spark before plan rewriting; only relevant when constant folding is disabled (#3448)
- [ ] `CurrentTimestamp` (`current_timestamp`) - typically constant-folded; see above
- [ ] `Now` (`now`) - typically constant-folded; see above
- [x] `CurrentTimeZone` (`current_timezone`)
- [x] `LocalTimestamp` (`localtimestamp`)
- [x] `DateDiff` (`datediff`, `date_diff`)
- [x] `DateAdd` (`date_add`, `dateadd`)
- [x] `DateFormatClass` (`date_format`)
- [x] `DateSub` (`date_sub`)
- [x] `DayOfMonth` (`day`, `dayofmonth`)
- [x] `DayOfYear` (`dayofyear`)
- [x] `FromUnixTime` (`from_unixtime`)
- [x] `FromUTCTimestamp` (`from_utc_timestamp`)
- [x] `Hour` (`hour`)
- [x] `LastDay` (`last_day`)
- [x] `Minute` (`minute`)
- [x] `Month` (`month`)
- [x] `MonthsBetween` (`months_between`) - #4417
- [x] `NextDay` (`next_day`)
- [x] `Quarter` (`quarter`)
- [x] `Second` (`second`)
- [ ] `ParseToTimestamp` (`to_timestamp`) - partial via #3467
- [ ] `ParseToTimestampNTZ` (`to_timestamp_ntz`)
- [ ] `ParseToTimestampLTZ` (`to_timestamp_ltz`)
- [ ] `ParseToDate` (`to_date`) - #3467
- [x] `ToUnixTimestamp` (`to_unix_timestamp`) - #4417
- [x] `ToUTCTimestamp` (`to_utc_timestamp`)
- [x] `TruncDate` (`trunc`)
- [x] `TruncTimestamp` (`date_trunc`)
- [x] `UnixTimestamp` (`unix_timestamp`)
- [x] `DayOfWeek` (`dayofweek`)
- [x] `WeekDay` (`weekday`)
- [ ] `DayName` (`dayname`) (3.5+)
- [x] `WeekOfYear` (`weekofyear`)
- [x] `Year` (`year`)
- [x] `MakeDate` (`make_date`)
- [x] `MakeTimestamp` (`make_timestamp`) - #4417
- [ ] `MakeTimestampNTZ` (`make_timestamp_ntz`) - #3108
- [ ] `MakeTimestampLTZ` (`make_timestamp_ltz`)
- [ ] `TryMakeTimestamp` (`try_make_timestamp`) - #3117
- [ ] `TryMakeTimestampNTZ` (`try_make_timestamp_ntz`)
- [ ] `TryMakeTimestampLTZ` (`try_make_timestamp_ltz`)
- [ ] `MonthName` (`monthname`) (4.0+)
- [x] `Extract` (`extract(field FROM source)`)
- [x] `DatePart` (`date_part`, `datepart`)
- [x] `DateFromUnixDate` (`date_from_unix_date`)
- [x] `UnixDate` (`unix_date`)
- [x] `SecondsToTimestamp` (`timestamp_seconds`)
- [x] `MillisToTimestamp` (`timestamp_millis`) - #4417
- [x] `MicrosToTimestamp` (`timestamp_micros`) - #4417
- [x] `UnixSeconds` (`unix_seconds`) - #4417
- [x] `UnixMillis` (`unix_millis`) - #4417
- [x] `UnixMicros` (`unix_micros`) - #4417
- [x] `ConvertTimezone` (`convert_timezone`)

## Interval construction

- [ ] `MakeInterval` (`make_interval`) - #3099
- [ ] `MakeDTInterval` (`make_dt_interval`) - #4338
- [ ] `MakeYMInterval` (`make_ym_interval`) - #3100
- [ ] `TryMakeInterval` - #3103

## Timestamp arithmetic (related catalyst expressions)

- [ ] `TimestampAdd` (`timestampadd`) - #3113
- [ ] `TimestampAddInterval` - #3114
- [ ] `TimestampAddYMInterval` - #3115
- [ ] `TimestampDiff` (`timestampdiff`) - #3116
- [ ] `SubtractDates` - #3094
- [ ] `SubtractTimestamps` - #3112
- [ ] `DateTimeSub` - #3134

## Time-window expressions

- [ ] `TimeWindow` (`window`)
- [ ] `SessionWindow` (`session_window`)
- [ ] `WindowTime` (`window_time`) - #3732 (via `PreciseTimestampConversion`)

## TimeType expressions (Spark 4.1+)

See umbrella issue #4288 for the full TimeType implementation plan.

- [x] `ToTime` (`to_time`) - default format only
- [x] `TryToTime` (`try_to_time`) - default format only
- [x] `MakeTime` (`make_time`)
- [ ] `CurrentTime` (`current_time`) - typically constant-folded
- [ ] `HoursOfTime` (`extract(HOUR FROM t)`) - #3126, #3615
- [ ] `MinutesOfTime` (`extract(MINUTE FROM t)`) - #3447
- [ ] `SecondsOfTime` (`extract(SECOND FROM t)`) - #3129, #3618
- [ ] `SecondsOfTimeWithFraction`
- [ ] `TimeFromSeconds` (`time_from_seconds`)
- [ ] `TimeFromMillis` (`time_from_millis`)
- [ ] `TimeFromMicros` (`time_from_micros`)
- [ ] `TimeToSeconds` (`time_to_seconds`)
- [ ] `TimeToMillis` (`time_to_millis`)
- [ ] `TimeToMicros` (`time_to_micros`)
- [ ] `TimeAddInterval`
- [ ] `SubtractTimes`
- [ ] `TimeDiff` (`time_diff`) - #3122
- [ ] `TimeTrunc` (`time_trunc`) - #3123

## Related work

- #4311 - Implement JVM UDFs for all date/time expressions (delivery strategy: route gaps through the codegen dispatcher for Spark-compatible defaults, native implementations remain opt-in)
- #4288 - TimeType support epic (covers infrastructure: shuffle, sort, min/max, casts)
- #3448 - Scalar inputs for datetime expressions when constant folding is disabled (covers `current_date`/`current_timestamp`/`now` regression cases)
- #2649 - `date_trunc` incorrect results in non-UTC timezone
- #3180 - `hour`/`minute`/`second` incorrect timezone handling for `TimestampNTZ`

### Describe the potential solution

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.