Validate format pattern in `dateTimeFieldSpecs` upon schema creation
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
The following error was encountered today
```
2021/11/03 14:21:49.226 INFO [LLRealtimeSegmentDataManager_table_realtime__0__0__20211101T1431Z] [table_realtime__0__0__20211101T1431Z] Waiting to acquire semaphore for building segment
2021/11/03 14:21:49.227 INFO [LLRealtimeSegmentDataManager_table_realtime__0__0__20211101T1431Z] [table_realtime__0__0__20211101T1431Z] Trying to build segment
2021/11/03 14:21:49.227 ERROR [LLRealtimeSegmentDataManager_table_realtime__0__0__20211101T1431Z] [table_realtime__0__0__20211101T1431Z] Could not build segment
java.lang.IllegalArgumentException: Illegal pattern component: T
at org.joda.time.format.DateTimeFormat.parsePatternTo(DateTimeFormat.java:566) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.joda.time.format.DateTimeFormat.createFormatterForPattern(DateTimeFormat.java:687) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.joda.time.format.DateTimeFormat.forPattern(DateTimeFormat.java:177) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.apache.pinot.spi.data.DateTimeFormatPatternSpec.(DateTimeFormatPatternSpec.java:57) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.apache.pinot.spi.data.DateTimeFormatSpec.(DateTimeFormatSpec.java:60) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig.setTime(SegmentGeneratorConfig.java:214) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig.(SegmentGeneratorConfig.java:140) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:83) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:794) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808
]
```
The relevant config was
```
"dateTimeFieldSpecs": [
{
"name": "DATETIME_F",
"dataType": "STRING",
"format": "1:SECONDS:SIMPLE_DATE_FORMAT:yyyy-MM-ddTHH:mm:ss",
"granularity": "1:SECONDS"
}
],
```
You can tell `yyyy-MM-ddTHH:mm:ss` should have been `yyyy-MM-dd'T'HH:mm:ss` instead. However, the error was not caught till we found the table had stopped consuming messages. Even further, and this format pattern is only validated when a segment is built, no before or during consumption.
Therefore, it would be benefit the users if we can fail fast and throw the error when creating the schema in the first place.
Contributor guide
Research direction
Start with DateTimeFormatSpec and SegmentGeneratorConfig, which appear in the reported stack trace, and trace how dateTimeFieldSpecs are handled during schema creation. Verify that the malformed pattern is rejected when the schema is created, before consumption reaches segment building.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100