segment/unavailable/count metric is misleading during handoff
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
Probably all versions.
### Description
The `segment/unavailable/count` metric is computed as the number of published segments that are not being served by historicals. But this is not the correct definition of "available": it should include realtime tasks as well. The effect is that the metric is misleading during handoff: it appears that unavailability spikes up before the new segments are loaded by historicals, even if all segments actually are continuously available on some combination of realtime tasks and historicals.
See https://druid.apache.org/docs/latest/design/architecture.html#indexing-and-handoff for definitions of terms that we should be using here.
Contributor guide
Research direction
Start with the metric computation and the architecture documentation on indexing and handoff linked in the issue. Trace how availability is determined for historicals and realtime tasks; done means segment/unavailable/count reflects segments unavailable from both task types and no longer spikes during handoff when segments remain continuously served.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100