apache / apache/gluten

Date type support

Open
#561 2 comments 0 reactions 0 assignees View on GitHub
enhancement velox backend
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

The date type is very complex when we offload to native library. Here is a summary we noted so far.


Spark
substrait
Arrow
Velox

date days since epoch
4 byte
date
Date32
Date

timestamp, ms since epoch
8 byte
timestamp_tz
Timestamp
Timestamp (16 bytes, ns)

YearMonthInterval



DayTimeInterval



- Spark:

1. Date: number of days from epoch. Date32
2. TimeStamp: micro second since epoch. 64bit
3. UnixTimeStamp: us since epoch. 64bit TimeStamp = UnixTimeStamp * 1000
4. Spark3.3?
- Parquet:
1. Date: int32, days since 1970-01-01
2. Time: a logical time type without a date with millisecond or microsecond precision
3. TimeStamp: a single int64 number that can be decoded into year, month, day, hour, minute, second and subsecond fields.
Two parameters:
isAdjustedToUTC must be either true or false.
precision must be one of MILLIS(ms), MICROS(us) or NANOS(ns). This list is subject to potential expansion in the future. Upon reading, unknown precision-s must be handled as unsupported features (rather than as errors in the data files).
- Arrow
1. Date64 msec since epoch
2. Date32 days since epoch
3. Timestamp same as parquet.
4. TimestampNTZ : no timezone
- Velox:
1. Timestamp, seconds + nano seconds (16byte)
2. Date type, Int32.
3. Velox's dwrf doesn't support date
4. Velox-> Arrow doesn't support date
Timestamp=> Time(nanoseconds)
Timestamp support in operators covers more than date.
5. Velox's parquet support. But expression support is limited

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.