apache / apache/pinot

[multistage] support broker segment pruning

Open
#18,201 3 comments 0 reactions 0 assignees View on GitHub
feature multi-stage
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

The default MSE query planner does not support [broker-side segment pruning](https://docs.pinot.apache.org/operate-pinot/tuning/segment-pruning) today. The physical optimizer (v2 planner) does support it, but it does not have full feature parity with MSE yet (https://github.com/apache/pinot/issues/15455).

We've noticed if one server becomes unhealthy through a gray failure, it can affect nearly all queries on the tenant since they fan out to all servers. Having pinot automatically route around unhealthy servers is being dealt with in https://github.com/apache/pinot/issues/16253, but ideal the query planner also avoids querying servers unnecessarily.

## Paths to support:
- [x] Non-partitioned leaf path: https://github.com/apache/pinot/pull/18237
- [ ] Partitioned leaf path
- [ ] Logical tables

## Additional features:
- [x] Short-circuit in broker if all segments are pruned; otherwise it fans out to all servers for intermediate stage: https://github.com/apache/pinot/pull/18538

cc @Jackie-Jiang @yashmayya

Contributor guide

Open the contributing guide

Research direction

Compare the default MSE query planner with the physical optimizer (v2 planner), then trace the remaining partitioned leaf and logical-table paths. Use the completed non-partitioned path and broker short-circuit work as context; done means broker-side segment pruning is supported for both remaining paths and avoids unnecessary server fanout.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.