Server & build telemetry — operational dashboard for runner health, build performance & cost

Start by confirming or enabling RunsOn v3 Flex's native CloudWatch dashboard and daily cost reports, then compare the available runner, build, capacity, and cost data with the requirements listed here. Document the missing telemetry and use that gap assessment to recommend a backend, v1 scope, retention window, and alert-routing approach.

Summary

Sibling to #321. That initiative tracks how each lecture series is configured (declarative, point-in-time, stored in-repo as JSON + Pages). This issue covers the other half: operational telemetry — how our builds and runners actually perform, behave, and cost over time. Different data shape (time-series, not config), different cadence (continuous, not on-change), different audience (ops, not authors), and — unlike #321 — it genuinely needs a managed time-series backend rather than a GitHub-native store. We keep the two planes separate, joined only by a one-way link (a series page in the status dashboard can link out to its telemetry board).

Relationship to #321 (the two-plane split)

Overlap is just build duration: telemetry owns the trend; #321 keeps at most a labelled snapshot value.

Goal

A single operational view ("is the fleet healthy, are builds fast, what are we spending?") that can pull from a range of services rather than being wired to one provider.

Services / data sources to support

What should be reported through this interface

A. Build / workflow — total + per-job duration (ci / cache / publish / collab), success & failure rate, failure classification (build error vs runner/infra vs cache miss vs dependency resolution), queue/wait time (trigger→pickup), re-run frequency, duration trend per series (regression detection after dep bumps).

B. Runner / host (RunsOn / EC2) — CPU, memory (peak + sustained), disk usage & IO (ties into the volume=80gb work), network IO, GPU utilization + GPU memory (g4dn — are we paying for GPU we don't use?), boot time, spot vs on-demand mix, spot interruptions, instance type / AMI in use.

C. Cost — $/build, $/day, $/repo, $/series, spot vs on-demand split, GPU vs CPU runner cost, idle/waste (instances up but not executing), month-to-date vs AWS budget.

D. Capacity & control-plane health (RunsOn Flex) — queue depth / backlog, unschedulable jobs, GitHub API rate-limit pressure, webhook latency / Lambda errors, RunsOn license status.

E. Alerting — unschedulable jobs, failure-rate spikes, cost/budget anomalies, stuck-runner / hard-limit timeouts → SNS → Slack/email.

Candidate tooling (no self-hosted webapp)

1. Start with RunsOn's native v3 Flex surfaces — CloudWatch stack dashboard + daily cost reports + inline per-job charts, zero new infra. Measure the gap first.
2. Unified pane: Grafana Cloud (free tier) fed by RunsOn OTLP (extras=otel + OtelExporterEndpoint) for the runner/EC2 side + Grafana GitHub data-source plugin for the cross-repo workflow layer.
3. Alternative: Amazon Managed Grafana ← CloudWatch to stay entirely inside AWS.

Open questions

• Backend: native RunsOn only, Grafana Cloud (OTLP), or Amazon Managed Grafana (CloudWatch)? Decide after evaluating native.
• v1 scope: RunsOn/EC2 only, or include the cross-repo GitHub Actions workflow layer from the start?
• Retention window (14 / 30 / 90 days)?
• GPU utilization via OTLP host metrics vs runs-on/action@v2 CloudWatch (bills per metric)?
• Alert routing + ownership (Slack? email? who's on point?).

Suggested first step

Confirm/enable RunsOn v3 Flex's native CloudWatch dashboard + daily cost reports, capture what's missing, then choose a backend.