getCompactionTaskCapacity is not worker category aware.
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Description
The [getCompactionTaskCapacity](https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java#L321-L329) function which is used to ascertain that the Druid cluster has enough task slots before the coordinator schedules additional compaction tasks doesn't take into consideration [overlord dynamic config](https://druid.apache.org/docs/latest/configuration/#overlord-dynamic-configuration). The overlord dynamic config can prevent compaction tasks from running on specific categories of workers. This way, the compaction task capacity is incorrectly overestimated.
For example, I have two worker categories, `compaction-category` with a total of `600` task slots and another `ingestion-category` with `2000` slots(high number because of multiple ingestion task replicas).
Using the overlord dynamic config,
- `compaction-category` is configured to run the following task types,
- kill
- compact
- single_phase_sub_task
- partial_dimension_cardinality
- partial_index_generate
- partial_index_generic_merge
- `ingestion-category` is configured to run,
- index_kafka
Now, [getCompactionTaskCapacity](https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java#L321-L329) would return `2600` as the total capacity, which is inaccurate since only `600` slots are actually available for compaction tasks. While this might not pose a problem in a healthy cluster, it becomes critical during compaction task failures. The oversight leads to the coordinator creating excessive compaction tasks, resulting in contention on compaction slots and slowing down all compaction tasks. This creates a feedback loop where the increasing number of compaction tasks exacerbates contention, ultimately overwhelming the overlord with too many tasks to handle
### Affected Version
Saw this on Druid 25. Is also present in master.
Contributor guide
Research direction
Start in server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java at getCompactionTaskCapacity, then read the Overlord dynamic configuration documentation linked in the issue. Trace how worker categories and permitted task types are represented, and verify that the reported capacity counts only slots eligible for compaction tasks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100