apache / apache/pinot

[multistage][feature] resource estimation and cost factory

Open
#12,322 1 comment 0 reactions 0 assignees View on GitHub
multi-stage stale
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Incident Recap
===
currently, we do not estimate how many resources / compute time is required to run a query.

This causes problems for the entire cluster when there's no strict resource isolations in-place.
- one expensive query could cause the entire cluster to be stuck in a non-responsive state
- what's worst, if any single-point-of-oom can occur it will occur (last stage sort or broker reduce)
- related issues: https://github.com/apache/pinot/issues/12309

even when we prevent these from happening post-mortem, the ideal state is to estimate and reject these types of queries

Problem Statement & Goal
===
a broader discussion for these is to create a cost/resource estimator, goal
1. provide query cost estimation without running the query (similar to query explain);
2. provide a hint for creating a cost-based query engine strategy

Design/Impl (TBD)
===
Most likely the easiest way is to go with the cost-factory plugin from Calcite.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing the problem statement, related issue #12309, and Calcite's cost-factory plugin; done means defining a design that estimates query cost without execution and supports cost-based query strategy hints.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.