Add "DateTime" DataType
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
Context
===
for more context. let's say user want to configure this during ingest:
```
"dateTimeFieldSpecs": [{
"name": "Date",
"dataType": "STRING",
"format" : "1:SECONDS:SIMPLE_DATE_FORMAT:MM/dd/yyyy HH:mm:ss a",
"granularity": "1:HOURS"
}]
```
they have to set the `"dataType"` to `STRING` because one want the result of
```
Select Date From myTable
```
to be a string that conforms with the SDF specified . (for example the STRING is directly feed into some downstream program)
Challenge
===
However,
1. in SQL database, setting a column to STRING type means we need to support >= and <= in the raw data format.
2. in Pinot, we cant support this SDF as time column format because they are not both lexical and time order consistent (e.g. `02/01/2021` comes after `01/29/2022` in string-ordering but before in timestamp-ordering), if we use this field as time field for partitioning real-time and offline table, we will have wrong results because the underlying ordering is STRING-based
3. one can also configure the `dataType` to `TIMESTAMP` and intrinsically convert to String in query, but the result has to be the ISO SQL standard yyyy-mm-ddTHH:MM:SS format, which might not be what the user wanted.
Problem Statement
===
We want to create some kind of ingestion configurable DataType (let's name it `DateTime`) that (1) returns a String that conforms with the ingestion configured SDF; and (2) ordered by EPOCH ordering;
So that
```
SELECT myDateTimeType FROM myTable ORDER BY myDateTimeType
```
returns
```
"02/01/2021 00:00:00"
"01/01/2022 00:00:00"
```
Solution
===
Create a DATE_TIME type, along with TIMESTAMP type they should be the only time column going forward.
- `TIMESTAMP` type is stored as LONG and returned as LONG or the ISO standard timestamp format.
- `DATE_TIME` type is stored as LONG and returned as the SDF format configured via the table DataSchema.
Alternative Solution
===
We can either store the actual data in STRING or LONG. but
1. if we store it in raw string format and force it to order by converted EPOCH, this requires us to convert it every time making a compare. very costly.
2. if we were to store it as LONG which is natively sorted in EPOCH, and only do the conversion when query: we need to store the original SDF configured by user during ingestion somewhere, so we need to find a way to let Pinot know during query time
Original Bug Report
===
Currently datetime objects are either stored as String with simple date format or stored as timestamp which persists as epoch long value.
This has several problems.
1. #7804 added validation that requires the underlying string and the simple date format needs to be both time and lexical ordering.
2. timestamp can only return a ISO standardized string
There's no way for users to request a customized stringify format return without it to be both time and lexical ordering. Propose to add a new DataType "DateTime". this data type - inserts and retrieves the column value as String according to the simple date format, but only order in epoch (e.g. cannot be used as String comparison).
This way we can support ingesting data and retrieving them in format such as `mm/dd/yyyy`.
Contributor guide
Research direction
The issue proposes a new DATE_TIME type alongside existing STRING and TIMESTAMP handling, using the table DataSchema for its configured format. Start by tracing those existing data types and the DataSchema, then identify relevant tests; done means the proposed type returns the configured format while preserving epoch ordering.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100