apache / apache/pinot

Ingest and query time in RFC3339

Open
#7,276 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Hi,

I'm having a time format in the Kafka Messages as follows:
```sh
"_time": "2021-08-10T07:08:08.873Z",
```

I'm using the following schema for the date:
```json
"dateTimeFieldSpecs": [
{
"name": "_time",
"dataType": "STRING",
"format": "1:MILLISECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
"granularity": "1:MINUTES"
}
],
```
(I'm using a Java format here)

How can I check that the ingest in Pinot is working with the specified date time field?

Then in a Presto SQL the DATETIMECONVERT doesn't work as expected and returns an error:
```sql
SELECT DATETIMECONVERT("_time", '1:MILLISECONDS:SIMPLE_DATE_FORMAT:yyyy-mm-ddTHH:MM:SS.%fZ', '1:DAYS:SIMPLE_DATE_FORMAT:yyyyMMdd', '1:DAYS'),
max(value_number) AS max_1
FROM abc.xyz
LIMIT 1000;
```
(Here I'm using a python format? as described in superset)

The returned error is as follows:
```sh
pinot error: must be real number, not dict
```

How can I use the column "_time" with the RFC 3339 format as the date time field in Presto / Superset?

Thanks for your help and cheers!

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied dateTimeFieldSpecs for _time and the DATETIMECONVERT query against the RFC3339 value. Check Pinot's documented date-time format and ingestion behavior, then verify whether the query succeeds and returns the requested day format without the reported error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.