canonical / canonical/observability
Inconsistent traffic flow from Grafana to Loki/Alertmanager/Prometheus
- Dominant language
- Just
- Stars
- 10
- Forks
- 25
- Avg merge
- 6h 3m
- Merged PRs (30d)
- 7
Description
When COS Lite is deployed, Grafana will have following data sources. As you see, there is an inconsistency in the traffic flow based on the destination services.
- To Alertmanager: http://192.168.124.138/observability-alertmanager (through Traefik)
- To Loki: http://192.168.124.138/observability-loki-0 (through Traefik)
- To Prometheus: http://prometheus-0.prometheus-endpoints.observability.svc.cluster.local:9090 (through K8s service endpoint)
```
$ juju status -m observability
Model Controller Cloud/Region Version SLA Timestamp
observability sunbeam-controller flying-oryx-k8s/localhost 3.6.23 unsupported 05:55:42Z
App Version Status Scale Charm Channel Rev Address Exposed Message
alertmanager 0.27.0 active 1 alertmanager-k8s 1/stable 180 10.152.183.85 no
catalogue active 1 catalogue-k8s 1/stable 87 10.152.183.72 no
grafana 9.5.21 active 1 grafana-k8s 1/stable 160 10.152.183.60 no
loki 2.9.6 active 1 loki-k8s 1/stable 199 10.152.183.179 no
prometheus 2.52.0 active 1 prometheus-k8s 1/stable 247 10.152.183.22 no
traefik 2.11.0 active 1 traefik-k8s latest/candidate 295 10.152.183.251 no Serving at http://192.168.124.138
Unit Workload Agent Address Ports Message
alertmanager/0* active idle 10.1.1.112
catalogue/0* active idle 10.1.2.161
grafana/0* active idle 10.1.2.139
loki/0* active idle 10.1.2.22
prometheus/0* active idle 10.1.1.19
traefik/0* active idle 10.1.1.190 Serving at http://192.168.124.138
Offer Application Charm Rev Connected Endpoint Interface Role
alertmanager-karma-dashboard alertmanager alertmanager-k8s 180 0/0 karma-dashboard karma_dashboard provider
grafana-dashboards grafana grafana-k8s 160 2/2 grafana-dashboard grafana_dashboard requirer
loki-logging loki loki-k8s 199 2/2 logging loki_push_api provider
prometheus-receive-remote-write prometheus prometheus-k8s 247 2/2 receive-remote-write prometheus_remote_write provider
prometheus-scrape prometheus prometheus-k8s 247 0/0 metrics-endpoint prometheus_scrape requirer
```
Contributor guide
Research direction
Reproduce the COS Lite deployment and inspect Grafana's configured data-source URLs alongside the Juju relations and `juju status -m observability` output. Compare the Alertmanager, Loki, and Prometheus paths through Traefik or Kubernetes service endpoints; done means their traffic routing is consistent and verified against the deployment status.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grafana, kubernetes, prometheus
- Domain
- infrastructure, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100