Segment replicas leading to split query views
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
We run Pinot 0.8.0. When setting up a realtime table consuming from a single-partition kafka topic, we set `replicasPerPartition` to 2, which means there are two consuming segments, running on two separate pinot servers. However, when you take one server down, wait for a while and then bring it back up, your query could still hit either of the two consuming segment although one is lagging behind, hence leading to inconsistent/incorrect query results. As a user, we expect to see the query gets routes to the consuming segment that has newer data.
Steps to reproduce:
1. Set up a realtime table with `replicasPerPartition` set to 2. Also set `realtime.segment.flush.threshold.time` to something like `12h` to make sure there is no segment flush during testing.
2. Have the 2 replica segments, distributed on Host A and B respectively, consume for 5 minutes.
3. Stop the pinot server on Host A. Wait for 5 minutes and then start it.
4. Wait for both consuming segments to come back online.
5. Run a PQL query like `select * from order by desc` that scans all segments in the UI repeatedly and you will see that the `numDocsScanned` alternates as the query gets routed to different consuming segments.
Contributor guide
Research direction
No source file or test is named. Reproduce the issue with a realtime table using replicasPerPartition=2 and the Host A failure sequence, then trace how queries choose between the returning consuming segments. Done means repeated queries consistently use the consuming segment with the newer data instead of alternating results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- database, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100