apache / apache/druid

[PROPOSAL] Add segment limit for native and streaming index tasks

Open
#7,238 5 comments 1 reaction 0 assignees View on GitHub
Area - Batch Ingestion Area - Streaming Ingestion Design Review Proposal
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Motivation

To prevent IndexTasks and Kafka/KinesisIndexTasks from consuming excessive memory, it would be safer if there was a configurable limit on the total # of segments that can be opened by a single task. An example of when a situation like this might occur is if an indexing task has a low segmentGranularity (e.g. an hour) and a backfilled/late data stream that has a small number of events per hour over a large number of hours. Without any safeguards, the task would try to open an excessive number of segments and OOM due to the overhead of opening files.

### Proposed changes

I plan on adding a `maxTotalSegments` field to `AppenderatorConfig` allowing users to set it in the ingestion spec.

| Property | Description | Default | Required? |
|------------------|----------------------------------------------------------------------------------------------|---------|-----------|
| maxTotalSegments | The maximum number of mutable segments that an indexing task is allowed to have open at one time. | 1000 | No |

If the number of mutable segments exceeds `maxTotalSegments` during indexing, the segments will be pushed to deep storage (`BatchAppenderatorDriver#pushAllAndClear()`). This behaviour is similar to what occurs if the number of rows exceeds `maxTotalRows`. For this to work, `AppenderatorDriverAddResult#isPushRequired()` should be modified to also check whether the number of segments is above the limit. This is feasible since the number of sinks (which have a 1:1 relation to mutable segments) is accessible from `AppenderatorImpl#add()` and can be returned as a new field in `AppenderatorDriverAddResult`.

### Rationale

I don't think there's any other options for capping the number of segments opened by an indexing task/supervisor.

### Operational impact

Addition of a field with a default to ingestion spec shouldn't have an operational impact.

Contributor guide

Open the contributing guide

Research direction

Start by reading AppenderatorConfig, AppenderatorDriverAddResult#isPushRequired(), AppenderatorImpl#add(), and BatchAppenderatorDriver#pushAllAndClear() to trace the existing maxTotalRows flow. Done means the ingestion spec can configure maxTotalSegments with the stated default and indexing tasks push and clear when the mutable-segment limit is exceeded.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.