canonical / canonical/cos-proxy-operator

Prometheus Scrape Job Cleanup

Open
#135 1 comment 0 reactions 0 assignees View on GitHub
Status: Triage Type: Bug
Dominant language
Python
Stars
3
Forks
12
PR merge metrics
No merged PRs in 30d

Description

### Bug Description

We have multiple cos-proxy applications deployed so that we can have them related to different instances of the prometheus-scrape-config-k8s so that we can have different scrape intervals (specifically for the alerts). Upon noticing that we had some redundant scrape jobs, I realized we had incorrectly related nrpe to both cos-proxy applications. However, removing the relation between nrpe and one of the cos-proxy applications did not clear the redundant scrape jobs.

When I removed the relation between cos-proxy and the prometheus-scrape-config-k8s, the redundant scrape jobs went away but when I re-added the relation, they came back.

Suspecting the cached info was being stored within the cos-proxy unit, I removed the cos-proxy unit. The redundant scrape jobs went away but when I added a new cos-proxy unit, they came back.

This led me to think the cached data must be held within one of the other cos-proxy relations. One at a time, I removed the other relations:

- ceph-dashboard:grafana-dashboard cos-proxy:dashboards
- cos-proxy:dashboards etcd:grafana
- cos-proxy:dashboards prometheus-libvirt-exporter:dashboards
- cos-proxy:dashboards prometheus-openstack-exporter:dashboards
- cos-proxy:prometheus-target prometheus-libvirt-exporter:scrap
- cos-proxy:prometheus-target prometheus-openstack-exporter:prometheus-openstack-exporter-service
- cos-proxy:juju-info grafana-agent-container:juju-info
- cos-proxy:downstream-grafana-dashboard cos-grafana:grafana-dashboard

After each removal, I checked the prometheus config and the redundant scrape jobs were still there.

Then, once again, I removed and re-added the cos-proxy/prometheus-scrape-config-k8s relationship and the redundant scrape jobs were gone.

I have to assume that removing one of the other relations successfully removed the cached data but that the config change didn't get pushed to prometheus until removing and re-adding the prometheus-scrape-config-k8s relation.

### To Reproduce

Although we discovered this because we have two cos-proxy applications, this can probably be replicated with just one. I haven't yet narrowed it down to a minimal set of relations but this requires a cos deployment and then in another model, cos-proxy with nrpe. Cause the nrpe unit to be removed and the scrape jobs associated with that unit will persist in prometheus config.

### Environment

juju 3.3.3 on maas 3.3.5

cos-proxy rev 71
nrpe rev 106
alertmanager-k8s rev 77
catalogue-k8s rev 19
grafana-k8s rev 82
loki-k8s rev 91
prometheus rev 129
prometheus-scrape-config-k8s rev 43
traefik-k8s rev 129

### Additional context

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with a COS deployment, cos-proxy, and nrpe, then inspect the Prometheus configuration after removing the nrpe unit and changing the listed relations. Compare whether scrape jobs persist after relation updates; done means stale scrape jobs disappear without removing and re-adding the prometheus-scrape-config-k8s relation.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, prometheus, python
Domain
devops, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.