canonical / canonical/observability-stack

Grafana uses Traefik ingress URL for in-model datasources and fails TLS verification with external CA

Open
#483 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HCL
Stars
9
Forks
12
Avg merge
2d 40m
Merged PRs (30d)
3

Description

Similar to https://github.com/canonical/observability/issues/454 and https://github.com/canonical/observability-stack/issues/395, In COS Lite deployment, Grafana is using the Traefik ingress endpoint for in-model datasource communication. When external TLS is configured on Traefik using a privately signed certificate, Grafana fails with `x509: certificate signed by unknown authority`. Providing the CA to Grafana through `manual-tls-certificates:trust_certificate` > `grafana:receive-ca-cert` resolves the issue.

**Env details:**

- Juju 3.6.28, COS Lite deployed on Kubernetes using juju bundle.
- External TLS is configured directly on Traefik using an externally generated certificate/key through config options.
- No Vault or certificate-provider charm is used for the Traefik certificate.
- Grafana, Prometheus, Loki, etc. are deployed in the same COS model.
- Grafana datasource is configured with the Traefik ingress URL.

**Expected Behaviour:**
1. Grafana should use the internal Prometheus service endpoint and avoid the external Traefik TLS path.
or
2. COS should provide a supported mechanism to propagate the external Traefik CA automatically to Grafana.

**Exported bundle (with workaround of using `manual-tls-certificates` charm):**

[cos-bundle.zip](https://github.com/user-attachments/files/31940756/cos-bundle.zip)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by inspecting the exported cos-bundle.zip and the Grafana datasource configuration in the COS Lite deployment. Trace how the Traefik ingress URL and the manual-tls-certificates trust_certificate to grafana:receive-ca-cert relation are configured. Done means Grafana can reach in-model datasources without failing TLS verification, either through an internal endpoint or supported CA propagation.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana, kubernetes
Domain
infrastructure, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.