apache / apache/pinot

User provided table configuration can crash servers

Open
#6,971 0 comments 3 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

# Context

Been brainstorming with @mayankshriv about my observations regarding Pinot stability. From running clusters in production and troubleshooting people in our community Slack, my takeaway is that it’s very easy to crash a Pinot Server if you a create tables for which your system is not resourced properly. OOM, segments beings too big or too numerous are all correlated to your input dataset and available compute resources.

It is very easy to shoot yourself in the foot and blow up your own cluster.

I'm opening this issue to brainstorm with the rest of the community and see how people think we could make Pinot more reliable from user provided settings.

# Proposal

Make Pinot self aware of its resource availability, introduce circuit-breakers and fail-fast mechanism to prevent such things and have Pinot operate within available resources.

This is certainly easier said than done. Making Pinot more reliable to user provided settings could be acheive with 2 combinable strategies:

1. Introduce an auto-tuned ingestion mode that can set ingestion settings (batch size, segment size, etc) based on available resources.
2. Introduce circuit breakers and fail-fast mechanism to prevent Pinot server to crash from OOM or FS overflows but still serve queries.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by turning the brainstorming proposal into a concrete scope around auto-tuned ingestion, circuit breakers, or fail-fast handling; done would require an agreed design and implementation that prevents resource-related server crashes while preserving query service.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.