[VL] Support timestamp on Velox backend
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
Phase 1:
- [x] Timestamp data conversions, including r2c, c2r etc.
- [x] Timestamp split
- [x] Timestamp Scan (TODO: support timestamp int96 type)
https://github.com/oap-project/velox/pull/205
Phase 2:
- [ ] Datetime functions in `spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala`
Note: For Foldable or Unevaluable expressions with no input, Spark's optimizer can precompute their values, allowing Gluten to treat them as constant columns.
- [ ] ConvertTimezone -- TimestampNTZ not supported
- [ ] CurrentTimeZone -- Unevaluable, compute by Spark
- [x] CurrentTimestamp -- Foldable, compute by Spark
- [ ] Extract -- A set of various datetime expressions. Not all expressions and type are supported.
- [x] FromUTCTimestamp
- [x] FromUnixTime
- [x] Hour
- [ ] LocalTimestamp -- Foldable, compute by Spark
- [ ] MakeInterval -- CalendarIntervalType not supported
- [x] MakeTimestamp
- [x] MakeTimestampLTZExpressionBuilder
- [ ] MakeTimestampNTZExpressionBuilder -- TimestampNTZType not supported
- [x] MicrosToTimestamp
- [x] MillisToTimestamp
- [x] Minute
- [ ] Now -- Foldable, compute by Spark
- [ ] ParseToTimestamp -- Result mismatch
- [ ] ParseToTimestampLTZExpressionBuilder -- Same as ParseToTimestamp
- [ ] ParseToTimestampNTZExpressionBuilder -- TimestampNTZType not supported
- [x] Second
- [x] SecondsToTimestamp -- https://github.com/facebookincubator/velox/pull/14222
- [x] ToUTCTimestamp
- [x] ToUnixTimestamp
- [x] TruncTimestamp -- https://github.com/facebookincubator/velox/pull/11340 & https://github.com/apache/incubator-gluten/pull/7611
- [ ] TryToTimestampExpressionBuilder -- Same as ParseToTimestamp
- [x] UnixMicros
- [x] UnixMillis
- [x] UnixSecond
- [x] UnixTimestamp
- Cast support
- Cast to Timestamp
- - Cast from date/integer/floating-point types to timestamp are supported.
- Cast from timestamp
- - Cast timestamp to date is supported.
- - Cast timestamp to string is supported with some inconsistency with Spark (see https://github.com/facebookincubator/velox/pull/11114 & https://github.com/facebookincubator/velox/pull/12630).
- - Cast timestamp to int is supported (https://github.com/facebookincubator/velox/pull/11468).
- Cast from/to TimestampNTZ
Phase 3:
- [ ] Enable operators including Filter/Project/Sort/Agg/Hash-partition Shuffle.
Contributor guide
Assessment
This issue has not been assessed yet.