canonical / canonical/observability-stack
Grafana uses Traefik ingress URL for in-model datasources and fails TLS verification with external CA
- Dominant language
- HCL
- Stars
- 9
- Forks
- 12
- Avg merge
- 2d 40m
- Merged PRs (30d)
- 3
Description
Similar to https://github.com/canonical/observability/issues/454 and https://github.com/canonical/observability-stack/issues/395, In COS Lite deployment, Grafana is using the Traefik ingress endpoint for in-model datasource communication. When external TLS is configured on Traefik using a privately signed certificate, Grafana fails with `x509: certificate signed by unknown authority`. Providing the CA to Grafana through `manual-tls-certificates:trust_certificate` > `grafana:receive-ca-cert` resolves the issue.
**Env details:**
- Juju 3.6.28, COS Lite deployed on Kubernetes using juju bundle.
- External TLS is configured directly on Traefik using an externally generated certificate/key through config options.
- No Vault or certificate-provider charm is used for the Traefik certificate.
- Grafana, Prometheus, Loki, etc. are deployed in the same COS model.
- Grafana datasource is configured with the Traefik ingress URL.
**Expected Behaviour:**
1. Grafana should use the internal Prometheus service endpoint and avoid the external Traefik TLS path.
or
2. COS should provide a supported mechanism to propagate the external Traefik CA automatically to Grafana.
**Exported bundle (with workaround of using `manual-tls-certificates` charm):**
[cos-bundle.zip](https://github.com/user-attachments/files/31940756/cos-bundle.zip)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by inspecting the exported cos-bundle.zip and the Grafana datasource configuration in the COS Lite deployment. Trace how the Traefik ingress URL and the manual-tls-certificates trust_certificate to grafana:receive-ca-cert relation are configured. Done means Grafana can reach in-model datasources without failing TLS verification, either through an internal endpoint or supported CA propagation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grafana, kubernetes
- Domain
- infrastructure, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100