[multistage] support broker segment pruning
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
The default MSE query planner does not support [broker-side segment pruning](https://docs.pinot.apache.org/operate-pinot/tuning/segment-pruning) today. The physical optimizer (v2 planner) does support it, but it does not have full feature parity with MSE yet (https://github.com/apache/pinot/issues/15455).
We've noticed if one server becomes unhealthy through a gray failure, it can affect nearly all queries on the tenant since they fan out to all servers. Having pinot automatically route around unhealthy servers is being dealt with in https://github.com/apache/pinot/issues/16253, but ideal the query planner also avoids querying servers unnecessarily.
## Paths to support:
- [x] Non-partitioned leaf path: https://github.com/apache/pinot/pull/18237
- [ ] Partitioned leaf path
- [ ] Logical tables
## Additional features:
- [x] Short-circuit in broker if all segments are pruned; otherwise it fans out to all servers for intermediate stage: https://github.com/apache/pinot/pull/18538
cc @Jackie-Jiang @yashmayya
Contributor guide
Research direction
Compare the default MSE query planner with the physical optimizer (v2 planner), then trace the remaining partitioned leaf and logical-table paths. Use the completed non-partitioned path and broker short-circuit work as context; done means broker-side segment pruning is supported for both remaining paths and avoids unnecessary server fanout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100