canonical / canonical/microceph
HostDown alert get triggered when having more than 4 nodes of microceph cluster
- Dominant language
- Go
- Stars
- 396
- Forks
- 74
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 7
Description
I have a 9 nodes microceph cluster deployed with microceph charm and added a relation with COS using opentelemetry-collector charm in between.
From the alerts list, I will see 6 of HostDown alerts for microceph, with the following description.
```
Host 'localhost:9283' is down, failed to scrape. VALUE = 0 LABELS = map[__name__:up instance:localhost:9283 job:microceph_0_default juju_application:microceph juju_model:openstack-machines juju_model_uuid:4ad70035-43e0-4de3-8a6b-9d793eb9b5fe juju_unit:microceph/1]
```
3 are UP, and these are the unit where ceph-mgr service is running which has prometheus module, so localhost:9283 is listening there.
However, other 6 microceph nodes doesn't have ceph-mgr service, so localhost:9283 is not listening, which is expected.
I believe this HostDown alert should be only for microceph unit where ceph-mgr is running, or different rule to be valid for all microceph units.
Contributor guide
Research direction
Start with the HostDown alert rule and the Prometheus scrape targets produced for the microceph application through the COS/opentelemetry-collector relation. Check how ceph-mgr units are distinguished from units without ceph-mgr, then verify that alerts are emitted only for valid targets while ceph-mgr units remain monitored.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, prometheus
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100