canonical / canonical/cos-proxy-operator
`cos-proxy:juju-info` creates a Vector alert rule without starting Vector
- Dominant language
- Python
- Stars
- 3
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
### Bug Description
We need to start the vector binary if we pass (over relation data) any alert rules requiring vector metrics. This is related to this issue:
- https://github.com/canonical/cos-proxy-operator/issues/216
The alert rule will eventually fire because the `vector_uptime_seconds` metric does not exist since the vector scrape job was not created on `juju-info` relation joined.
### To Reproduce
1. deploy cos-proxy `rev166` # without any relations
2. `jssh cos-proxy/0 "ps -aux | grep vector"` # vector is not running
3. `jrel nrpe:general-info cos-proxy`
4. `jssh cos-proxy/0 "ps -aux | grep vector"` # vector is not running
5. deploy prom, offer its `metrics-endpoint` and consume it in the LXD model
6. `jrel cos-proxy:downstream-prometheus-scrape prom`
7. `app_data prom/0 metrics-endpoint` # see the vector alert rule
```json
{
"alert_rules": "{\"groups\": [{\"name\": \"test_3_84e570a3_cos_proxy_vector_restarted_alerts\", \"rules\": [{\"alert\": \"VectorRestarted\", \"expr\": \"vector_uptime_seconds < (vector_uptime_seconds offset 5m)\", \"for\": \"0m\", \"labels\": {\"severity\": \"info\", \"juju_model\": \"test-3\", \"juju_model_uuid\": \"84e570a3-2d74-4f5a-878a-23c4c5673d0f\", \"juju_application\": \"cos-proxy\", \"juju_charm\": \"cos-proxy\"}, \"annotations\": {\"summary\": \"Vector restarted (instance {{ $labels.instance }})\", \"description\": \"Vector has just been restarted, less than one minute ago on {{ $labels.instance }}. Telemetry loss may have occurred.\\nVALUE = {{ $value }}\\nLABELS = {{ $labels }}\\n\"}}]}, {\"name\": \"test_3_84e570a3_cos_proxy_HostHealth_alerts\", \"rules\": [{\"alert\": \"HostDown\", \"expr\": \"up < 1\", \"for\": \"5m\", \"labels\": {\"severity\": \"critical\", \"juju_model\": \"test-3\", \"juju_model_uuid\": \"84e570a3-2d74-4f5a-878a-23c4c5673d0f\", \"juju_application\": \"cos-proxy\", \"juju_charm\": \"cos-proxy\"}, \"annotations\": {\"summary\": \"Host '{{ $labels.instance }}' is down.\", \"description\": \"Host '{{ $labels.instance }}' is down, failed to scrape.\\n VALUE = {{ $value }}\\n LABELS = {{ $labels }}\"}}, {\"alert\": \"HostMetricsMissing\", \"expr\": \"absent(up)\", \"for\": \"5m\", \"labels\": {\"severity\": \"critical\", \"juju_model\": \"test-3\", \"juju_model_uuid\": \"84e570a3-2d74-4f5a-878a-23c4c5673d0f\", \"juju_application\": \"cos-proxy\", \"juju_charm\": \"cos-proxy\"}, \"annotations\": {\"summary\": \"Metrics not received from host '{{ $labels.instance }}', failed to remote write.\", \"description\": \"Metrics not received from host '{{ $labels.instance }}', failed to remote write.\\n VALUE = {{ $value }}\\n LABELS = {{ $labels }}\"}}]}]}",
"scrape_jobs": "[]"
}
```
---
### LXD bundle
```yaml
default-base: ubuntu@22.04/stable
saas:
prom:
url: microk8s:admin/prom.prom
applications:
cos-proxy:
charm: cos-proxy
channel: 2/edge
revision: 166
num_units: 1
to:
- "0"
constraints: arch=amd64
trust: true
nrpe:
charm: nrpe
channel: latest/stable
revision: 221
trust: true
machines:
"0":
constraints: arch=amd64
relations:
- - nrpe:general-info
- cos-proxy:juju-info
- - cos-proxy:downstream-prometheus-scrape
- prom:metrics-endpoint
```
### k8s bundle
```yaml
bundle: kubernetes
saas:
remote-7b2b2e0c990b4c078eb29ffc204bc5ce: {}
applications:
prom:
charm: prometheus-k8s
channel: dev/edge
revision: 278
resources:
prometheus-image: 152
scale: 1
constraints: arch=amd64
storage:
database: kubernetes,1,1024M
trust: true
relations:
- - prom:metrics-endpoint
- remote-7b2b2e0c990b4c078eb29ffc204bc5ce:downstream-prometheus-scrape
--- # overlay.yaml
applications:
prom:
offers:
prom:
endpoints:
- metrics-endpoint
acl:
admin: admin
```
Contributor guide
No contributing guide indexed for this repository
Research direction
The reproduction uses the cos-proxy:juju-info relation and the Vector process; start by tracing relation handling after nrpe:general-info is joined and inspect how alert_rules and scrape_jobs are produced. Done means Vector starts when the relation supplies Vector-dependent rules and the resulting data includes the required scrape job.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus, python
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100