apache / apache/pinot

Throttle Ingestion in case of server capacity issues

Open
#9,547 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently, there's no safeguard in place to slow down or stop ingestion (realtime or offline) if any one of the Pinot servers are running out of disk space. Having this feature would allow us to prevent server crashes due to ingestion spikes.

CC @jadami10 @Jackie-Jiang

Contributor guide

Open the contributing guide

Research direction

Start by tracing Pinot’s realtime and offline ingestion entry points and how server disk capacity is exposed. Clarify the thresholds and behavior for throttling or stopping ingestion when any server is near capacity. Done should prevent ingestion spikes from causing server crashes, with coverage for both ingestion modes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.