apache / apache/pinot

After segment purge, the segment start and end time in segment zk metadata may not reflect the correct values

Open
#10,951 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

During the process of purging segments based on a record matcher, certain rows may be eliminated. Once the segment is rebuilt, new metadata is generated. However, in certain scenarios, the time metadata from the previous segment is carried over to the new segment. Consequently, even if rows from the beginning of the segments are removed, the segment start time metadata will still reflect the start time of the segment before purge. The same applies to the segment end time metadata.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the segment purge and rebuild flow that produces segment ZK metadata, focusing on how start and end times are carried from the previous segment. Reproduce a purge that removes rows at either boundary and inspect the resulting metadata; done means both values reflect the rebuilt segment's remaining rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.