SDF conversion mismatch between segment completion and broker routing
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
Once a segment is completed, either converted from consuming to online in real-time or uploaded in offline, the start/end time in zk metadata is updated based on the raw data.
For SDF, start/end time is stored in millis, and the conversion is done by DateTimeFormatter (default time zone: local).
```
DateTimeFormatter formatter = DateTimeFormat.forPattern("yyyyMMdd");
formatter.parseMillis("20170701"); -> 1498892400000
```
For broker time boundary management, the time millis are converted back to SDF to compute the time boundary. For broker time segment pruner, the query time condition is parsed to time millis, and compared with segment metadata. These conversions are done by TimeFormatSpec (default time zone: UTC).
```
DateTimeFormatSpec dateTimeFormatSpec = new DateTimeFormatSpec("1:DAYS:SIMPLE_DATE_FORMAT:yyyyMMdd");
dateTimeFormatSpec.fromFormatToMillis("20170701") -> 1498867200000
```
If Pinot clusters are in time zones other than UTC, the time boundary will be shifted accordingly, and the time pruner will filter wrong segments instead of expected.
We should unify the conversion to avoid this. One approach is to standardize this to always use UTC timezone, but it won’t be backward compatible. Another approach is to keep both default time zones in local, this will require all Pinot instances to be in the same local time zone.
Contributor guide
Research direction
Compare the DateTimeFormatter conversion used when completed segments update zk metadata with TimeFormatSpec in broker time boundary management and the broker segment pruner. Reproduce the differing millis values for SDF dates in a non-UTC timezone, then define a timezone strategy that preserves or deliberately handles existing metadata and makes boundary and pruning results consistent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100