[agentrx-optimizer] Daily Workflow Optimization - 2026-09-14

### Executive Summary

AgentRx analyzed 80 scheduled runs of **PR Sous Chef** (`.github/workflows/pr-sous-chef.lock.yml`) from the last 24h, reconstructed as trajectories from GitHub Actions job/step telemetry (`run_summary.json` + `jobs.json`), since raw log text for individual steps was not readable in this sandbox (permission-restricted). 37 of 80 runs (46%) failed, and **35 of those 37 (95%) fail at the exact same step**: `Check daily workflow token guardrail` in the `activation` job. This step failing causes the `agent` job to be skipped entirely (condition never met), so the workflow never even attempts its task — the failure happens before any agent work starts.

Fleet-wide, this single step is also the single largest concentrated failure pattern in the whole 24h window: 35 of 112 total failed runs across every workflow (31%) are this one step, and 100% of those are PR Sous Chef.

Root-cause analysis of the guardrail script (`actions/setup/js/check_daily_aic_workflow_guardrail.cjs`) shows this is not the intended "over budget" outcome — it's a **fail-closed error-handling gap**: any error while scanning prior runs' AI-credit usage (rate limit, missing/expired usage artifact, transient API failure) unconditionally calls `core.setFailed()`, even though the code deliberately avoids `setFailed()` for the adjacent "budget exceeded" case with a comment explaining that a graceful skip is the correct behavior.

### AgentRx Evidence

- **Critical step:** `Check daily workflow token guardrail` (step id `daily-effective-workflow-guardrail`), `activation` job, step ``#7`` — [source](../../blob/main/actions/setup/js/check_daily_aic_workflow_guardrail.cjs)
- **Failure category:** Fail-closed error handling — `transient_error`/`structural_error` paths call `core.setFailed()` unconditionally in the catch block (lines ~653–662), unlike the sibling "exceeded" path a few lines above which intentionally logs and returns without failing.
- **Frequency / impact:** 35 of 80 PR Sous Chef runs (44%) over a continuous ~11-hour window (2026-09-13 05:40–16:30 UTC); 35 of 112 fleet-wide failures (31%) in the 24h sample. Every occurrence skips the entire `agent` job — 100% of attempted work lost on each occurrence, not just degraded quality.
- **Representative run IDs:** first occurrence [§34740895922](https://github.com/github/gh-aw/actions/runs/34740895922) (05:40 UTC), mid-streak [§34754028110](https://github.com/github/gh-aw/actions/runs/34754028110) (11:16 UTC), last occurrence [§34768779630](https://github.com/github/gh-aw/actions/runs/34768779630) (16:30 UTC), recovery [§34769320963](https://github.com/github/gh-aw/actions/runs/34769320963) (16:41 UTC, success).

AgentRx Artifacts

**Pipeline stages run:** `ir` (completed), `static`/`dynamic`/`judge` (failed — see Limitations).

**IR summary:** 80 trajectories built from `run_summary.json`/`jobs.json` job-and-step sequences (one event per GitHub Actions step, ordered by job: `pre_activation` → `activation` → `agent` → `conclusion`). All 80 trajectories parsed as valid IR (`agentrx/runs/gh-aw-daily/trajectory_ir.json`). 37 trajectories terminate with the `activation` job step sequence: `Restore daily AIC scan observations` (success) → `Check daily workflow token guardrail` (**failure**) → all subsequent activation steps skipped → `agent`/`detection`/`safe_outputs` jobs skipped → `conclusion` job success (workflow overall: failure).

**Manual invariant check (in place of the `check`/`judge` stages — see Limitations):**

| violation | evidence | fix_type | rationale |
| --- | --- | --- | --- |
| Guardrail scan errors hard-fail the whole scheduled run | 35/80 runs: `activation` step `Check daily workflow token guardrail` = failure → `agent` job skipped → workflow conclusion = failure | improving retry/backoff strategy (fail-open on transient error) | `check_daily_aic_workflow_guardrail.cjs` catch block calls `core.setFailed()` for both `transient_error` and `structural_error`, but the code's own comment states budget-exceeded is intentionally non-failing; scan errors (rate limit, missing artifact) are transient by nature and shouldn't be treated the same as a hard misconfiguration |
| Agent job never runs when guardrail scan errors | `agent`, `detection`, `safe_outputs`, `evals` jobs all report `conclusion: skipped` in every one of the 35 failing runs | reducing wasted job scheduling | Confirms 100% of intended work (not just guardrail accounting) is lost per occurrence — this is the true blast radius, not merely a "red X" |

**Known limitations:** The `static`, `dynamic`, and `judge` AgentRx stages require an authenticated Copilot CLI endpoint (`COPILOT_GITHUB_TOKEN`/`GH_TOKEN`), which is not available in this sandbox — those stages failed immediately with "No authentication information found" (reported separately via `missing_tool`). Only the `ir` stage completed; the invariant table above was produced by direct inspection of the structured job/step data and the guardrail script source rather than the LLM-based `check`/`judge` stages. Individual GitHub Actions step log text (`job-*.log`) was also unreadable (root-owned, `EACCES`), so the classification is based on step-conclusion telemetry and source code, not raw stdout.

### Recommended Optimization

**Change:** In `actions/setup/js/check_daily_aic_workflow_guardrail.cjs`, in the `main()` catch block (~line 653), stop calling `core.setFailed(message)` for `transient_error` (rate limits, retryable API failures) — instead log a warning and return, the same fail-open pattern already used a few lines above for the `exceeded` status. Reserve `core.setFailed()` for genuine `structural_error` cases (404/401 — missing workflow, bad token) where operator intervention is actually required.

**Why this is highest impact:** This single change would have prevented 35 of 80 (44%) of PR Sous Chef's runs from failing outright, and it's the largest single failure cluster across the entire fleet in this 24h window (31% of all failures). It requires no new tooling — it aligns the error path with a pattern the codebase already implements correctly for the sibling "budget exceeded" case, so it's a small, low-risk, surgical fix rather than a new feature.

**Where to implement:** `actions/setup/js/check_daily_aic_workflow_guardrail.cjs`, lines ~653–662 (the `catch (error)` block in `main()`). The corresponding compiled step is generated by `pkg/workflow/compiler_activation_daily_aic.go::buildActivationDailyAICGuardrailStep` — no compiler change should be needed since the compiler just invokes the script; only the script's error handling needs adjustment. After editing, all `.lock.yml` files must be recompiled via the standard gh-aw compile step.

### Validation Plan

- Recompile and let PR Sous Chef run on its next few scheduled triggers; confirm the `activation` job now completes with `conclusion: success` (or its own graceful skip) even when the guardrail scan hits a transient error, and that the `agent` job is no longer unconditionally skipped in that scenario.
- Track the `Check daily workflow token guardrail` step's failure rate for PR Sous Chef over the next 24h — expect it to drop from 44% toward 0%, with any remaining failures limited to genuine `structural_error` cases (bad token/missing workflow).
- Fleet-wide: expect total failed-run count across all workflows to drop by roughly 30% (the current share attributable to this one step), since the fix applies to every workflow using `max-daily-ai-credits`, not just PR Sous Chef.

### References

- [§34740895922](https://github.com/github/gh-aw/actions/runs/34740895922) — first failure in the streak (2026-09-13 05:40 UTC)
- [§34768779630](https://github.com/github/gh-aw/actions/runs/34768779630) — last failure before recovery (2026-09-13 16:30 UTC)
- [§34769320963](https://github.com/github/gh-aw/actions/runs/34769320963) — first successful run after recovery (2026-09-13 16:41 UTC)

> [!WARNING]
>
> Firewall blocked 1 domain
>
> The following domain was blocked by the firewall during workflow execution:
>
> - `api.anthropic.com`
>
> To allow these domains, add them to the `network.allowed` list in your workflow frontmatter:
>
> ```yaml
> network:
> allowed:
> - defaults
> - "api.anthropic.com"
> ```
>
> See [Network Configuration](https://github.github.com/gh-aw/reference/network/) for more information.
>
>

> Generated by [⚡ Daily AgentRx Trace Optimizer](https://github.com/github/gh-aw/actions/runs/34809983330) · claude · agent · 290.7 AIC · ⌖ 8.48 AIC · ⊞ 8.8K · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+is%3Aissue+%22gh-aw-workflow-call-id%3A+github%2Fgh-aw%2Fdaily-agentrx-trace-optimizer%22&type=issues)
> - [x] expires on Sep 20, 2026, 10:03 PM UTC-08:00

Start in actions/setup/js/check_daily_aic_workflow_guardrail.cjs, especially the main() catch block around lines 653–662, and compare it with the nearby non-failing budget-exceeded path. Preserve failures for structural errors while making transient scan errors non-fatal, then recompile the .lock.yml files using the standard gh-aw compile step. Done means transient guardrail errors no longer fail the activation job or skip the agent job.

Written by the indexing model from the issue text.