apache / apache/pinot

SDF conversion mismatch between segment completion and broker routing

Open
#6,548 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

Once a segment is completed, either converted from consuming to online in real-time or uploaded in offline, the start/end time in zk metadata is updated based on the raw data.
For SDF, start/end time is stored in millis, and the conversion is done by DateTimeFormatter (default time zone: local).
```
DateTimeFormatter formatter = DateTimeFormat.forPattern("yyyyMMdd");
formatter.parseMillis("20170701"); -> 1498892400000
```

For broker time boundary management, the time millis are converted back to SDF to compute the time boundary. For broker time segment pruner, the query time condition is parsed to time millis, and compared with segment metadata. These conversions are done by TimeFormatSpec (default time zone: UTC).

```
DateTimeFormatSpec dateTimeFormatSpec = new DateTimeFormatSpec("1:DAYS:SIMPLE_DATE_FORMAT:yyyyMMdd");
dateTimeFormatSpec.fromFormatToMillis("20170701") -> 1498867200000
```

If Pinot clusters are in time zones other than UTC, the time boundary will be shifted accordingly, and the time pruner will filter wrong segments instead of expected.

We should unify the conversion to avoid this. One approach is to standardize this to always use UTC timezone, but it won’t be backward compatible. Another approach is to keep both default time zones in local, this will require all Pinot instances to be in the same local time zone.

Contributor guide

Open the contributing guide

Research direction

Compare the DateTimeFormatter conversion used when completed segments update zk metadata with TimeFormatSpec in broker time boundary management and the broker segment pruner. Reproduce the differing millis values for SDF dates in a non-UTC timezone, then define a timezone strategy that preserves or deliberately handles existing metadata and makes boundary and pruning results consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.