compiler-explorer / compiler-explorer/infra

Manage Grafana Cloud alerting/dashboards with Terraform

Open
#2,314 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
434
Forks
429
Avg merge
4d 21h
Merged PRs (30d)
38

Description

*(Filed by Claude on behalf of @mattgodbolt)*

Our Grafana Cloud alert rules, contact points and dashboards are hand-configured in the UI, so they are undocumented, unreviewed and easy to lose. Today's example: an alert on `changes(process_start_time_seconds{job="compiler_explorer"}[30m]) > 0` (app restarted by systemd, see #2313 and compiler-explorer/compiler-explorer#9035) was clicked together by hand. A `/nosym/tmp` / root free-space alert (`node_filesystem_avail_bytes{mountpoint="/"}`) from #2310 is the obvious next one.

The `grafana/grafana` Terraform provider covers `grafana_folder`, `grafana_rule_group`, `grafana_contact_point`, `grafana_notification_policy`, `grafana_dashboard`. Rough plan:

- Create a Grafana Cloud service account + token with alerting/dashboard scopes; store in SSM next to `promPassword`/`lokiPassword`.
- Add the provider to `terraform/` (or a sibling state to keep blast radius separate from AWS), import the existing folder/contact points/rules, and move the hand-made rules into code.
- First rules: app-restart (above), root filesystem free space per host, maybe the Papertrail ingest-quota proxy.
- Note the Grafana metrics side is already IaC-ish via `grafana/agent.yaml` + `install-agent.sh`; this is about the server-side config.

Separate workload from the incident follow-ups; tracking here.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing terraform/ configuration and grafana/agent.yaml with install-agent.sh to distinguish server-side Grafana settings from metrics collection. Then review how promPassword and lokiPassword are stored in SSM, and map the existing Grafana folders, contact points, rules, and dashboards to the provider resources named in the issue. Done means the selected alerting and dashboard configuration is managed in Terraform without conflating this work with the incident follow-ups.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, grafana, terraform
Domain
infrastructure, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.