apache / apache/pinot

Add sanity check to the output of Realtime Provisioning Tool

Open
#8,339 8 comments 0 reactions 0 assignees View on GitHub
good first issue
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

Currently RTP tool generates a mutable segments based on the given schema characteristics (or a given sample segment) and then linearly extrapolates the required memory, segment size, etc for different combinations of the input parameters.
We've recently seen people have used 5GB segments because of the tool's suggestion. It's fine if the tool outputs 5GB because of the chosen input parameters, but it should give some warnings about potential problematic values.
Here are some suggestions:
- Change the output data structure to include an array of warnings
- If a suggested segment size is > 500MB, add a warning message that the desired segment size should be < 500MB.
- If number of segments is greater than a threshold (maybe 5K?), then the tools add a warning for that.
- Sometimes tools outputs NA if the memory goes above the provided availableHostMemory parameter. Instead of NA, we should output the actual value, but add a warning that memory for some combinations goes above available host memory

Contributor guide

Open the contributing guide

Research direction

Locate the Realtime Provisioning Tool's output-generation entry point and inspect how suggested segment size, segment count, and memory values are produced. Confirm the warning thresholds, then verify that the output includes warnings, reports actual memory values instead of NA when host memory is exceeded, and flags potentially problematic segment sizes and segment counts.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.