Throttle Ingestion in case of server capacity issues
Open
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Currently, there's no safeguard in place to slow down or stop ingestion (realtime or offline) if any one of the Pinot servers are running out of disk space. Having this feature would allow us to prevent server crashes due to ingestion spikes.
CC @jadami10 @Jackie-Jiang
Contributor guide
Research direction
Start by tracing Pinot’s realtime and offline ingestion entry points and how server disk capacity is exposed. Clarify the thresholds and behavior for throttling or stopping ingestion when any server is near capacity. Done should prevent ingestion spikes from causing server crashes, with coverage for both ingestion modes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100