apache / apache/pinot

[multistage] Support for Serving Multi-Stage Query from the Same Replica Group

Open
#9,325 0 comments 0 reactions 0 assignees View on GitHub
multi-stage
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

There are a few use-cases where we'd like to serve a query for all tables involved in a multi-stage query from the same replica-group. Examples:

1. Colocated Join
2. Self-Join for Upsert Tables (strictReplicaGroup routing won't work since BaseInstanceSelector uses an auto-incrementing requestId)

One rough idea to support this is:

1. Pass Pinot Broker requestId to BaseInstanceSelector instead of using its own auto-incrementing requestId.
2. To pass requestId from Broker Request Handlers to the BaseInstanceSelector, add requestId to BrokerRequest.

cc: @walterddr @Jackie-Jiang @yupeng9

Contributor guide

Open the contributing guide

Research direction

Start by tracing how requestId moves through the Broker Request Handlers and how BaseInstanceSelector currently generates and uses its own requestId. Read the BrokerRequest path and the strictReplicaGroup behavior, then verify the intended design against colocated joins and self-joins for upsert tables. Done means all tables in a multi-stage query can be served from the same replica group without breaking existing routing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.