Metric taskSlot/idle/count doesn't exclude disabled workers
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
29.0.1
### Description
I've noticed that metric `taskSlot/idle/count` that is part of `TaskSlotCountStatsMonitor` is not excluding slots from disabled workers.
I'm running overlord in httpRemote mode and it seems like this metric should account for it - it calls for [getWorkersEligibleToRunTasks()](https://github.com/apache/druid/blob/15b3efd085d3e69417a00d06b04e325fd26e729a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java#L1728) which checks if the worker [isEnabled()](https://github.com/apache/druid/blob/15b3efd085d3e69417a00d06b04e325fd26e729a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java#L375), but then when I connect to the middlemanager instance I get:
```
$ curl -X POST localhost:8091/druid/worker/v1/disable
{"druid-middlemanager-default-0.druid-middlemanager.druid-test.svc.cluster.local:8091":"disabled"}
$ curl -s localhost:8000/metrics | grep slot | grep .0
druid_middlemanager_worker_taskslot_total_count{category="__default_worker_category_",druid_service="druid/middleManager",} 3.0
druid_middlemanager_worker_taskslot_used_count{category="__default_worker_category_",druid_service="druid/middleManager",} 0.0
druid_middlemanager_worker_taskslot_idle_count{category="__default_worker_category_",druid_service="druid/middleManager",} 3.0
```

So in the end I'm kinda confused why the metric doesn't show proper value - I guess I'm missing something while reading the code..?
Contributor guide
Research direction
Start with TaskSlotCountStatsMonitor and the linked getWorkersEligibleToRunTasks() and isEnabled() paths in HttpRemoteTaskRunner.java. Reproduce the disabled-worker scenario using the shown disable request and metrics query, then trace how eligible workers feed taskSlot/idle/count. Done means the metric reflects the expected value when a worker is disabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100