[multistage] broker/planner query logic tracker
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Currently, multi-stage engine doesn't support many of the broker pre-processing logic, we should support:
- [x] REALTIME/OFFLINE split support (https://github.com/apache/pinot/pull/9379)
- [ ] broker side pruning logic support
- [ ] tenant and tenant isolation support
in addition, many of the post-aggregation logic should also be consolidated:
- [ ] consolidate post aggregate function logic between intermediate stage and data table reducer
- [ ] make broker execute intermediate stage as well (https://github.com/apache/pinot/issues/9368)
Larger design initiatives
- [ ] cost-based optimization
- [ ] pluggable query plan optimizer (rule and hints)
- [ ] #9611
Contributor guide
Research direction
Start by reviewing the completed REALTIME/OFFLINE split pull request #9379 and the linked intermediate-stage issue #9368, then examine the remaining broker preprocessing and post-aggregation items. The issue is a broad tracker, so work is complete only after a specific unchecked initiative is selected, implemented, and validated with the project's relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100