Upgrade Grafana

Summary

Upgrade the Grafana Helm chart from 7.0.14 to 8.5.9, which ships Grafana 11.3.0. This version includes a fix for unbounded growth of the alert_rule_version table.

We've been burned by this bug when this table fills up and prevents updates to alerts from being applied. The workaround to is to run TRUNCATE TABLE alert_rule_version; in the Grafana database.

Required Changes

1. Fix assertNoLeakedSecrets failure (hard blocker)

Chart 8.x introduced assertNoLeakedSecrets: true, which rejects any grafana.ini value containing a secret-like pattern. Our current implementation passes the Postgres connection string (with credentials)
directly in the grafana.ini database block (aws_eks_cluster.py).

The database URL must be moved to a Kubernetes Secret, mounted as an env var (GF_DATABASE_URL), and referenced in grafana.ini using Grafana's variable expansion:

 ini                                                                                                                                                                                                                
[database]      
url = ${GF_DATABASE_URL}

2. Verify service account token mounting

Chart 8.x changed the default for service account token auto-mounting from true to false. We don't explicitly set this, so it will flip on upgrade. Verify Grafana operates correctly without the token, or
explicitly set:

serviceAccount:
  automountServiceAccountToken: true

3. Confirm no legacy alerting settings in cluster

Grafana 11 hard-fails to start if any legacy alerting settings are present. Verify no such settings exist in the cluster before deploying.

4. Assess username case normalization impact

Grafana 11 normalizes all usernames to lowercase. Users who logged in with mixed-case usernames via auth.proxy may see a new account created on first login, losing dashboard preferences and org roles. Assess
whether any real users are affected before rolling out.

Out of Scope

• No changes needed to alerting provisioning format — contact points, policies, and templates (apiVersion: 1) are still valid
• No changes needed to sidecar.alerts / sidecar.dashboards config

Test Plan

• Deploy to a non-production control room and confirm Grafana starts successfully
• Verify alerting contact points and notification policies are intact
• Verify dashboards load and sidecar picks up ConfigMaps correctly
• Verify auth proxy login works and existing users retain their roles

Start with aws_eks_cluster.py and the Grafana Helm chart configuration, then review the chart 8.x defaults for secret handling and service account token mounting. Deploy to a non-production control room and verify startup, alerting, dashboards, sidecar ConfigMaps, auth-proxy logins, and existing user roles. Done means the listed checks pass without legacy alerting settings or exposed database credentials.