apache / apache/pinot

[multistage] broker/planner query logic tracker

Open
#9,165 0 comments 0 reactions 0 assignees View on GitHub
multi-stage
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently, multi-stage engine doesn't support many of the broker pre-processing logic, we should support:

- [x] REALTIME/OFFLINE split support (https://github.com/apache/pinot/pull/9379)
- [ ] broker side pruning logic support
- [ ] tenant and tenant isolation support

in addition, many of the post-aggregation logic should also be consolidated:
- [ ] consolidate post aggregate function logic between intermediate stage and data table reducer
- [ ] make broker execute intermediate stage as well (https://github.com/apache/pinot/issues/9368)

Larger design initiatives
- [ ] cost-based optimization
- [ ] pluggable query plan optimizer (rule and hints)
- [ ] #9611

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the completed REALTIME/OFFLINE split pull request #9379 and the linked intermediate-stage issue #9368, then examine the remaining broker preprocessing and post-aggregation items. The issue is a broad tracker, so work is complete only after a specific unchecked initiative is selected, implemented, and validated with the project's relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.