canonical / canonical/observability

Inconsistent traffic flow from Grafana to Loki/Alertmanager/Prometheus

Open
#454 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Just
Stars
10
Forks
25
Avg merge
6h 3m
Merged PRs (30d)
7

Description

When COS Lite is deployed, Grafana will have following data sources. As you see, there is an inconsistency in the traffic flow based on the destination services.

- To Alertmanager: http://192.168.124.138/observability-alertmanager (through Traefik)
- To Loki: http://192.168.124.138/observability-loki-0 (through Traefik)
- To Prometheus: http://prometheus-0.prometheus-endpoints.observability.svc.cluster.local:9090 (through K8s service endpoint)

Image

```
$ juju status -m observability
Model Controller Cloud/Region Version SLA Timestamp
observability sunbeam-controller flying-oryx-k8s/localhost 3.6.23 unsupported 05:55:42Z

App Version Status Scale Charm Channel Rev Address Exposed Message
alertmanager 0.27.0 active 1 alertmanager-k8s 1/stable 180 10.152.183.85 no
catalogue active 1 catalogue-k8s 1/stable 87 10.152.183.72 no
grafana 9.5.21 active 1 grafana-k8s 1/stable 160 10.152.183.60 no
loki 2.9.6 active 1 loki-k8s 1/stable 199 10.152.183.179 no
prometheus 2.52.0 active 1 prometheus-k8s 1/stable 247 10.152.183.22 no
traefik 2.11.0 active 1 traefik-k8s latest/candidate 295 10.152.183.251 no Serving at http://192.168.124.138

Unit Workload Agent Address Ports Message
alertmanager/0* active idle 10.1.1.112
catalogue/0* active idle 10.1.2.161
grafana/0* active idle 10.1.2.139
loki/0* active idle 10.1.2.22
prometheus/0* active idle 10.1.1.19
traefik/0* active idle 10.1.1.190 Serving at http://192.168.124.138

Offer Application Charm Rev Connected Endpoint Interface Role
alertmanager-karma-dashboard alertmanager alertmanager-k8s 180 0/0 karma-dashboard karma_dashboard provider
grafana-dashboards grafana grafana-k8s 160 2/2 grafana-dashboard grafana_dashboard requirer
loki-logging loki loki-k8s 199 2/2 logging loki_push_api provider
prometheus-receive-remote-write prometheus prometheus-k8s 247 2/2 receive-remote-write prometheus_remote_write provider
prometheus-scrape prometheus prometheus-k8s 247 0/0 metrics-endpoint prometheus_scrape requirer
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the COS Lite deployment and inspect Grafana's configured data-source URLs alongside the Juju relations and `juju status -m observability` output. Compare the Alertmanager, Loki, and Prometheus paths through Traefik or Kubernetes service endpoints; done means their traffic routing is consistent and verified against the deployment status.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana, kubernetes, prometheus
Domain
infrastructure, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.