apache / apache/druid

Metric taskSlot/idle/count doesn't exclude disabled workers

Open
#16,771 1 comment 0 reactions 0 assignees View on GitHub
Area - Metrics/Event Emitting
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

29.0.1

### Description

I've noticed that metric `taskSlot/idle/count` that is part of `TaskSlotCountStatsMonitor` is not excluding slots from disabled workers.
I'm running overlord in httpRemote mode and it seems like this metric should account for it - it calls for [getWorkersEligibleToRunTasks()](https://github.com/apache/druid/blob/15b3efd085d3e69417a00d06b04e325fd26e729a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java#L1728) which checks if the worker [isEnabled()](https://github.com/apache/druid/blob/15b3efd085d3e69417a00d06b04e325fd26e729a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java#L375), but then when I connect to the middlemanager instance I get:
```
$ curl -X POST localhost:8091/druid/worker/v1/disable
{"druid-middlemanager-default-0.druid-middlemanager.druid-test.svc.cluster.local:8091":"disabled"}

$ curl -s localhost:8000/metrics | grep slot | grep .0
druid_middlemanager_worker_taskslot_total_count{category="__default_worker_category_",druid_service="druid/middleManager",} 3.0
druid_middlemanager_worker_taskslot_used_count{category="__default_worker_category_",druid_service="druid/middleManager",} 0.0
druid_middlemanager_worker_taskslot_idle_count{category="__default_worker_category_",druid_service="druid/middleManager",} 3.0
```

![image](https://github.com/user-attachments/assets/7e16ec86-665a-46c0-aa38-17a39979e7c3)

So in the end I'm kinda confused why the metric doesn't show proper value - I guess I'm missing something while reading the code..?

Contributor guide

Open the contributing guide

Research direction

Start with TaskSlotCountStatsMonitor and the linked getWorkersEligibleToRunTasks() and isEnabled() paths in HttpRemoteTaskRunner.java. Reproduce the disabled-worker scenario using the shown disable request and metrics query, then trace how eligible workers feed taskSlot/idle/count. Done means the metric reflects the expected value when a worker is disabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.