apache / apache/pinot

Infer SDF for Timestamp in dimension field spec

Open
#8,117 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

When ingesting a `Timestamp` field as a String that has an SDF (Simple Date Format) pattern that is non-default for `java.sql.Timestamp` in a schema's dimension field spec (which requires `yyyy-mm-dd hh:mm:ss[.fffffffff]`) Pinot will fail to import the record. To work around this, the source data must either be modified to match the standard SDF or transformed in the table spec.

Example:

```json
"dimensionFieldSpecs": [{
"name": "Date",
"dataType": "TIMESTAMP"
}]
```

Source data:

```csv
ID, DATE
1, "10/1/2005"
```

The SDF format here is not specified, since time granularity is not allowed as a dimension. The same issue can be experienced for ingesting a String field as a `Timestamp` in the date field specification of a schema, which does allow a user to specify an SDF. There are some scenarios where using the dimension field specification to automatically convert from a String in the source dataset to a `Timestamp` is useful and simpler than using the date time field spec.

The solution to this issue is to allow for conversions to a `Timestamp` from a String as an input that automatically infers popular SDFs that already have a pre-defined granularity.

Contributor guide

Open the contributing guide

Research direction

Start by tracing Pinot's schema dimension-field ingestion path for String-to-Timestamp conversion, then compare it with the date field specification's SDF handling. The work is done when dimension fields can infer supported popular SDFs for Timestamp input and the examples in the issue import successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.