apache / apache/pinot

Segment replicas leading to split query views

Open
#7,850 12 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

We run Pinot 0.8.0. When setting up a realtime table consuming from a single-partition kafka topic, we set `replicasPerPartition` to 2, which means there are two consuming segments, running on two separate pinot servers. However, when you take one server down, wait for a while and then bring it back up, your query could still hit either of the two consuming segment although one is lagging behind, hence leading to inconsistent/incorrect query results. As a user, we expect to see the query gets routes to the consuming segment that has newer data.

Steps to reproduce:
1. Set up a realtime table with `replicasPerPartition` set to 2. Also set `realtime.segment.flush.threshold.time` to something like `12h` to make sure there is no segment flush during testing.
2. Have the 2 replica segments, distributed on Host A and B respectively, consume for 5 minutes.
3. Stop the pinot server on Host A. Wait for 5 minutes and then start it.
4. Wait for both consuming segments to come back online.
5. Run a PQL query like `select * from order by desc` that scans all segments in the UI repeatedly and you will see that the `numDocsScanned` alternates as the query gets routed to different consuming segments.

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Reproduce the issue with a realtime table using replicasPerPartition=2 and the Host A failure sequence, then trace how queries choose between the returning consuming segments. Done means repeated queries consistently use the consuming segment with the newer data instead of alternating results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
database, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.