[multistage][feature] resource estimation and cost factory
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
Incident Recap
===
currently, we do not estimate how many resources / compute time is required to run a query.
This causes problems for the entire cluster when there's no strict resource isolations in-place.
- one expensive query could cause the entire cluster to be stuck in a non-responsive state
- what's worst, if any single-point-of-oom can occur it will occur (last stage sort or broker reduce)
- related issues: https://github.com/apache/pinot/issues/12309
even when we prevent these from happening post-mortem, the ideal state is to estimate and reject these types of queries
Problem Statement & Goal
===
a broader discussion for these is to create a cost/resource estimator, goal
1. provide query cost estimation without running the query (similar to query explain);
2. provide a hint for creating a cost-based query engine strategy
Design/Impl (TBD)
===
Most likely the easiest way is to go with the cost-factory plugin from Calcite.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by reviewing the problem statement, related issue #12309, and Calcite's cost-factory plugin; done means defining a design that estimates query cost without execution and supports cost-based query strategy hints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100