[multistage] Support for Serving Multi-Stage Query from the Same Replica Group
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
There are a few use-cases where we'd like to serve a query for all tables involved in a multi-stage query from the same replica-group. Examples:
1. Colocated Join
2. Self-Join for Upsert Tables (strictReplicaGroup routing won't work since BaseInstanceSelector uses an auto-incrementing requestId)
One rough idea to support this is:
1. Pass Pinot Broker requestId to BaseInstanceSelector instead of using its own auto-incrementing requestId.
2. To pass requestId from Broker Request Handlers to the BaseInstanceSelector, add requestId to BrokerRequest.
cc: @walterddr @Jackie-Jiang @yupeng9
Contributor guide
Research direction
Start by tracing how requestId moves through the Broker Request Handlers and how BaseInstanceSelector currently generates and uses its own requestId. Read the BrokerRequest path and the strictReplicaGroup behavior, then verify the intended design against colocated joins and self-joins for upsert tables. Done means all tables in a multi-stage query can be served from the same replica group without breaking existing routing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100