[PROPOSAL] Add segment limit for native and streaming index tasks
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Motivation
To prevent IndexTasks and Kafka/KinesisIndexTasks from consuming excessive memory, it would be safer if there was a configurable limit on the total # of segments that can be opened by a single task. An example of when a situation like this might occur is if an indexing task has a low segmentGranularity (e.g. an hour) and a backfilled/late data stream that has a small number of events per hour over a large number of hours. Without any safeguards, the task would try to open an excessive number of segments and OOM due to the overhead of opening files.
### Proposed changes
I plan on adding a `maxTotalSegments` field to `AppenderatorConfig` allowing users to set it in the ingestion spec.
| Property | Description | Default | Required? |
|------------------|----------------------------------------------------------------------------------------------|---------|-----------|
| maxTotalSegments | The maximum number of mutable segments that an indexing task is allowed to have open at one time. | 1000 | No |
If the number of mutable segments exceeds `maxTotalSegments` during indexing, the segments will be pushed to deep storage (`BatchAppenderatorDriver#pushAllAndClear()`). This behaviour is similar to what occurs if the number of rows exceeds `maxTotalRows`. For this to work, `AppenderatorDriverAddResult#isPushRequired()` should be modified to also check whether the number of segments is above the limit. This is feasible since the number of sinks (which have a 1:1 relation to mutable segments) is accessible from `AppenderatorImpl#add()` and can be returned as a new field in `AppenderatorDriverAddResult`.
### Rationale
I don't think there's any other options for capping the number of segments opened by an indexing task/supervisor.
### Operational impact
Addition of a field with a default to ingestion spec shouldn't have an operational impact.
Contributor guide
Research direction
Start by reading AppenderatorConfig, AppenderatorDriverAddResult#isPushRequired(), AppenderatorImpl#add(), and BatchAppenderatorDriver#pushAllAndClear() to trace the existing maxTotalRows flow. Done means the ingestion spec can configure maxTotalSegments with the stated default and indexing tasks push and clear when the mutable-segment limit is exceeded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100