Infer SDF for Timestamp in dimension field spec
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
When ingesting a `Timestamp` field as a String that has an SDF (Simple Date Format) pattern that is non-default for `java.sql.Timestamp` in a schema's dimension field spec (which requires `yyyy-mm-dd hh:mm:ss[.fffffffff]`) Pinot will fail to import the record. To work around this, the source data must either be modified to match the standard SDF or transformed in the table spec.
Example:
```json
"dimensionFieldSpecs": [{
"name": "Date",
"dataType": "TIMESTAMP"
}]
```
Source data:
```csv
ID, DATE
1, "10/1/2005"
```
The SDF format here is not specified, since time granularity is not allowed as a dimension. The same issue can be experienced for ingesting a String field as a `Timestamp` in the date field specification of a schema, which does allow a user to specify an SDF. There are some scenarios where using the dimension field specification to automatically convert from a String in the source dataset to a `Timestamp` is useful and simpler than using the date time field spec.
The solution to this issue is to allow for conversions to a `Timestamp` from a String as an input that automatically infers popular SDFs that already have a pre-defined granularity.
Contributor guide
Research direction
Start by tracing Pinot's schema dimension-field ingestion path for String-to-Timestamp conversion, then compare it with the date field specification's SDF handling. The work is done when dimension fields can infer supported popular SDFs for Timestamp input and the examples in the issue import successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100