apache / apache/druid

getCompactionTaskCapacity is not worker category aware.

Open
#15,847 2 comments 0 reactions 0 assignees View on GitHub
Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description

The [getCompactionTaskCapacity](https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java#L321-L329) function which is used to ascertain that the Druid cluster has enough task slots before the coordinator schedules additional compaction tasks doesn't take into consideration [overlord dynamic config](https://druid.apache.org/docs/latest/configuration/#overlord-dynamic-configuration). The overlord dynamic config can prevent compaction tasks from running on specific categories of workers. This way, the compaction task capacity is incorrectly overestimated.

For example, I have two worker categories, `compaction-category` with a total of `600` task slots and another `ingestion-category` with `2000` slots(high number because of multiple ingestion task replicas).

Using the overlord dynamic config,
- `compaction-category` is configured to run the following task types,
- kill
- compact
- single_phase_sub_task
- partial_dimension_cardinality
- partial_index_generate
- partial_index_generic_merge

- `ingestion-category` is configured to run,
- index_kafka

Now, [getCompactionTaskCapacity](https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java#L321-L329) would return `2600` as the total capacity, which is inaccurate since only `600` slots are actually available for compaction tasks. While this might not pose a problem in a healthy cluster, it becomes critical during compaction task failures. The oversight leads to the coordinator creating excessive compaction tasks, resulting in contention on compaction slots and slowing down all compaction tasks. This creates a feedback loop where the increasing number of compaction tasks exacerbates contention, ultimately overwhelming the overlord with too many tasks to handle

### Affected Version
Saw this on Druid 25. Is also present in master.

Contributor guide

Open the contributing guide

Research direction

Start in server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java at getCompactionTaskCapacity, then read the Overlord dynamic configuration documentation linked in the issue. Trace how worker categories and permitted task types are represented, and verify that the reported capacity counts only slots eligible for compaction tasks.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.