apache / apache/druid

Coordinator is not calulating compaction taskSlots correctly

Open
#16,694 2 comments 0 reactions 0 assignees View on GitHub
Area - Metrics/Event Emitting
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Druid is on 25.0.0 running on kubernetes.

I see one issue in auto-compaction for coordinator
when it calculates the totalSlots - It doesn't take into account the middle-manager-tiers.
We have 2 tiers.

for example:
1. compact_tier(10 MMs with 4 slots each)
2. default_tier(50 MMs with 10 slots each)
So It ignores tiers and shows maxCompactionslots to 54 which is [0.1*(40+500)]

In this log it shows wrong
`"Found [32] available task slots for compaction out of [54] max compaction task capacity"`

Contributor guide

Open the contributing guide

Research direction

Start at the Coordinator's auto-compaction task-slot calculation and inspect how maximum compaction capacity is derived from middle-manager tiers. Reproduce the two-tier setup described—10 workers with 4 slots and 50 workers with 10 slots—and verify that the logged available and maximum values account for both tiers.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.