apache / apache/pinot

Validate format pattern in `dateTimeFieldSpecs` upon schema creation

Open
#7,689 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

The following error was encountered today

```
2021/11/03 14:21:49.226 INFO [LLRealtimeSegmentDataManager_table_realtime__0__0__20211101T1431Z] [table_realtime__0__0__20211101T1431Z] Waiting to acquire semaphore for building segment
2021/11/03 14:21:49.227 INFO [LLRealtimeSegmentDataManager_table_realtime__0__0__20211101T1431Z] [table_realtime__0__0__20211101T1431Z] Trying to build segment
2021/11/03 14:21:49.227 ERROR [LLRealtimeSegmentDataManager_table_realtime__0__0__20211101T1431Z] [table_realtime__0__0__20211101T1431Z] Could not build segment
java.lang.IllegalArgumentException: Illegal pattern component: T
at org.joda.time.format.DateTimeFormat.parsePatternTo(DateTimeFormat.java:566) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.joda.time.format.DateTimeFormat.createFormatterForPattern(DateTimeFormat.java:687) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.joda.time.format.DateTimeFormat.forPattern(DateTimeFormat.java:177) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.apache.pinot.spi.data.DateTimeFormatPatternSpec.(DateTimeFormatPatternSpec.java:57) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.apache.pinot.spi.data.DateTimeFormatSpec.(DateTimeFormatSpec.java:60) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig.setTime(SegmentGeneratorConfig.java:214) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig.(SegmentGeneratorConfig.java:140) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:83) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:794) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808
]
```

The relevant config was
```
"dateTimeFieldSpecs": [
{
"name": "DATETIME_F",
"dataType": "STRING",
"format": "1:SECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-ddTHH:mm:ss",
"granularity": "1:SECONDS"
}
],
```
You can tell `yyyy-MM-ddTHH:mm:ss` should have been `yyyy-MM-dd'T'HH:mm:ss` instead. However, the error was not caught till we found the table had stopped consuming messages. Even further, and this format pattern is only validated when a segment is built, no before or during consumption.

Therefore, it would be benefit the users if we can fail fast and throw the error when creating the schema in the first place.

Contributor guide

Open the contributing guide

Research direction

Start with DateTimeFormatSpec and SegmentGeneratorConfig, which appear in the reported stack trace, and trace how dateTimeFieldSpecs are handled during schema creation. Verify that the malformed pattern is rejected when the schema is created, before consumption reaches segment building.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.