apache / apache/pinot

Add a new broker routing strategy to handle Realtime table query results consistency issue

Open
#15,727 4 comments 0 reactions 0 assignees View on GitHub
real-time stale
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

For a realtime table with >=2 replicas, the broker routing may hit any CONSUMING segment, which might cause user side to see inconsistent results(results may not be constantly increasing) due to the consuming speed for each server.

To implement this, we can:
1. From controller side, define a mapping for every table with the lead(master) partition to server mapping.
2. From broker side, build the new routing strategy to honor this lead partition to instance mapping for the CONSUMING segment.

Contributor guide

Open the contributing guide

Research direction

Begin in the controller and broker paths for realtime tables, focusing on how CONSUMING segments are selected. Trace the existing replica-routing behavior before designing the lead-partition-to-server mapping. Done means queries consistently honor the lead partition’s instance mapping for these segments, with the relevant behavior verified by the project's tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.