[cao-maintenance:reliability] Control-plane reliability requires attention

GitHub App installation-token API rate limits repeatedly blocked the CAO Activity collector during the last 24h (2026-09-09 23:32 → 2026-09-10 23:32 UTC), causing the shared `cao-activity` cache to miss on every subsequent run. Because several downstream jobs set `fail-on-cache-miss: true` when restoring that cache, the rate-limit incident cascaded into at least 8 separate `CAO Dashboard Build` failures across ~3 hours, blocking dashboard refresh and degrading control-plane observability during that window. This is a shared CAO infrastructure defect, not an isolated target-repo failure.

**Action:** Maintainer should have Copilot harden the activity-cache restore path to fail soft (skip/report-incomplete) instead of hard-failing the whole job on a rate-limited activity refresh, and verify the fix with a rerun of `CAO Activity` and `CAO Dashboard Build` after the next scheduled activity refresh succeeds.

## Failure clusters (window: 2026-09-09T23:32Z–2026-09-10T23:32Z)

1. **Activity collector rate-limited (root cause)** — 2 occurrences
- `CAO Activity / 34533473841` (21:40 UTC): `gh: API rate limit exceeded for installation ... (HTTP 403)`, job `index` exits 1, activity JSONL cache save skipped (`Path Validation Error: ... does not exist`).
- `CAO Activity / 34538573859` (22:40 UTC): identical rate-limit error, same failure mode.
2. **Downstream cache-miss cascade (fail-closed but blocking)** — 8 occurrences
- `CAO Dashboard Build` runs at 20:46, 20:56, 21:29, 21:52 (×2), 23:04, 23:11, 23:12, 23:18 UTC all fail with: `##[error]Failed to restore cache entry. Exiting as fail-on-cache-miss is set. Input key: cao-activity-v3-lookup--1`.
- Two later `CAO Dashboard Build` runs (23:24, 23:31 UTC) succeeded once the activity cache repopulated, confirming the cache-miss cascade self-resolves but still causes multi-hour build gaps.

## Assessment

- These are genuine control-plane infrastructure incidents, not intentional policy denials or target-owned task failures: the activity refresh and dashboard build are both CAO-owned shared jobs, and the failures block the control plane's ability to keep dashboard data current for ~3 hours per incident.
- `CAO reliability`/`CAO efficiency` review runs at 22:30 UTC also failed with an unrelated `Cannot find module '/tmp/gh-aw/actions/otlp.cjs'` OTLP export error; this appears to be a single transient tooling glitch (2 occurrences, same timestamp, no repeat since) and is noted for awareness but not counted as part of this cluster pending further recurrence.

Representative run references

- Rate limit root cause: https://github.com/githubnext/gh-aw-cao/actions/runs/34533473841 , https://github.com/githubnext/gh-aw-cao/actions/runs/34538573859
- Cache-miss cascade (sample): https://github.com/githubnext/gh-aw-cao/actions/runs/34541585576 , https://github.com/githubnext/gh-aw-cao/actions/runs/34540485059
- Recovery confirmation: https://github.com/githubnext/gh-aw-cao/actions/runs/34542001587 (success), https://github.com/githubnext/gh-aw-cao/actions/runs/34542509263 (success)

Collection method and limitations

Evidence gathered via GitHub Actions REST API (`list_workflow_runs` filtered by `status=failure` and `created > 2026-09-09T23:32:00Z`, ~28 total repository failures in window) and `get_job_logs` on the CAO-named runs. Coverage limited to the 24h evidence window; earlier recurrence not checked. No shared activity-cache artifact was available locally in this worker run, so all evidence was collected directly via bounded Actions API queries rather than the cache.

Agent prompt

The CAO activity collector workflow (likely `.github/workflows/activity.yml` and/or `.github/workflows/shared/activity-cache.md`) hits `gh: API rate limit exceeded for installation` during its `index` job, which prevents it from writing `gh-aw-logs.jsonl` and therefore prevents `actions/cache/save` from succeeding (`Path Validation Error: Path(s) specified ... do not exist`). Downstream consumers (e.g. dashboard build workflow) restore this same cache with `fail-on-cache-miss: true`, so any single rate-limited activity run causes every dashboard build dispatched before the next successful activity refresh to hard-fail with `Failed to restore cache entry. Exiting as fail-on-cache-miss is set.`

Please:
1. Add retry/backoff (or a rate-limit-aware early exit that still writes a minimal valid JSONL/cache entry) to the activity collector's `gh` calls in the `index` job, so a transient installation rate limit doesn't leave the cache in a state with no artifact to save.
2. Change dashboard-build (and any other consumer) cache restores that currently use `fail-on-cache-miss: true` against `cao-activity-v3-*` keys to fail soft: skip the dependent step or fall back to the previous cache generation via `restore-keys`, and report an incomplete/degraded status rather than hard-failing the whole job.
3. Validate by re-running `CAO Activity` and confirming a subsequent `CAO Dashboard Build` succeeds even when simulating a stale/missing activity cache (e.g. by temporarily invalidating the cache key in a test run).

Affected files to investigate: `.github/workflows/activity.yml`, `.github/workflows/shared/activity-cache.md`, and the dashboard-build workflow definitions that restore the `cao-activity-v3-*` cache with `fail-on-cache-miss: true`.

### Control Plane
- Correlation ID: 34542383992-2
- Central repository: githubnext/gh-aw-cao
- Control plane run: https://github.com/githubnext/gh-aw-cao/actions/runs/34542383992

> Generated by [CAO Maintenance / Reliability](https://github.com/githubnext/gh-aw-cao/actions/runs/34542383992) · copilot · auto · 84.8 AIC · ⌖ 9.96 AIC · ⊞ 11.9K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-cao+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-cao%2Fcao-maintenance-reliability%22&type=issues)
> - [x] expires on Sep 24, 2026, 11:40 PM UTC

Start with .github/workflows/activity.yml and .github/workflows/shared/activity-cache.md, then locate dashboard-build workflow definitions restoring cao-activity-v3-* with fail-on-cache-miss: true. Review the CAO Activity index job and the referenced failed runs, then validate by rerunning CAO Activity and confirming CAO Dashboard Build succeeds with a stale or missing activity cache without blocking the whole job.

Written by the indexing model from the issue text.