apache / apache/pinot

Add protection to Pinot upon reload of table

Open
#11,562 1 comment 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

I experience this issue upon configuring my star tree index. I moved a high cardinality col from the top of the dimensionSplitOrder to the bottom which 3x my star tree. Upon reload of the table, my cluster faced high CPU and memory which killed my cluster. I couldn't stop the reload of the segments and the only way is to disable my table. It could be good if Pinot could stop the reload with warning/error messages telling me that reload has stopped as my star tree 3x in size.

Contributor guide

Open the contributing guide

Research direction

Reproduce the scenario by moving a high-cardinality column in the star tree dimensionSplitOrder, enlarging the star tree, and reloading the table. Investigate the table reload and star tree configuration paths; done means an unsafe reload stops and reports a warning or error before resource exhaustion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.