Ingest and query time in RFC3339
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Hi,
I'm having a time format in the Kafka Messages as follows:
```sh
"_time": "2021-08-10T07:08:08.873Z",
```
I'm using the following schema for the date:
```json
"dateTimeFieldSpecs": [
{
"name": "_time",
"dataType": "STRING",
"format": "1:MILLISECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
"granularity": "1:MINUTES"
}
],
```
(I'm using a Java format here)
How can I check that the ingest in Pinot is working with the specified date time field?
Then in a Presto SQL the DATETIMECONVERT doesn't work as expected and returns an error:
```sql
SELECT DATETIMECONVERT("_time", '1:MILLISECONDS:SIMPLE_DATE_FORMAT:yyyy-mm-ddTHH:MM:SS.%fZ', '1:DAYS:SIMPLE_DATE_FORMAT:yyyyMMdd', '1:DAYS'),
max(value_number) AS max_1
FROM abc.xyz
LIMIT 1000;
```
(Here I'm using a python format? as described in superset)
The returned error is as follows:
```sh
pinot error: must be real number, not dict
```
How can I use the column "_time" with the RFC 3339 format as the date time field in Presto / Superset?
Thanks for your help and cheers!
Contributor guide
Research direction
Start by reproducing the supplied dateTimeFieldSpecs for _time and the DATETIMECONVERT query against the RFC3339 value. Check Pinot's documented date-time format and ingestion behavior, then verify whether the query succeeds and returns the requested day format without the reported error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100