Date type support
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
The date type is very complex when we offload to native library. Here is a summary we noted so far.
Spark
substrait
Arrow
Velox
date days since epoch
4 byte
date
Date32
Date
timestamp, ms since epoch
8 byte
timestamp_tz
Timestamp
Timestamp (16 bytes, ns)
YearMonthInterval
DayTimeInterval
- Spark:
1. Date: number of days from epoch. Date32
2. TimeStamp: micro second since epoch. 64bit
3. UnixTimeStamp: us since epoch. 64bit TimeStamp = UnixTimeStamp * 1000
4. Spark3.3?
- Parquet:
1. Date: int32, days since 1970-01-01
2. Time: a logical time type without a date with millisecond or microsecond precision
3. TimeStamp: a single int64 number that can be decoded into year, month, day, hour, minute, second and subsecond fields.
Two parameters:
isAdjustedToUTC must be either true or false.
precision must be one of MILLIS(ms), MICROS(us) or NANOS(ns). This list is subject to potential expansion in the future. Upon reading, unknown precision-s must be handled as unsupported features (rather than as errors in the data files).
- Arrow
1. Date64 msec since epoch
2. Date32 days since epoch
3. Timestamp same as parquet.
4. TimestampNTZ : no timezone
- Velox:
1. Timestamp, seconds + nano seconds (16byte)
2. Date type, Int32.
3. Velox's dwrf doesn't support date
4. Velox-> Arrow doesn't support date
Timestamp=> Time(nanoseconds)
Timestamp support in operators covers more than date.
5. Velox's parquet support. But expression support is limited
Contributor guide
Assessment
This issue has not been assessed yet.