githubnext / githubnext/gh-aw-cao

[cao-evolution:reliability] SelfCare orchestrator fails every scheduled run with Copilot CLI HTTP 400 on first turn

Closed
#11,936 0 comments 0 reactions 0 assignees View on GitHub
cao-evolution cao-evolution:reliability
Dominant language
JavaScript
Stars
3
Forks
1
Avg merge
49m
Merged PRs (30d)
837

Description

The `SelfCare · scheduled` orchestrator workflow failed in **all 7 scheduled runs** in the last 24 hours (2026-09-15 22:30 UTC – 2026-09-16 01:43 UTC). Every failure has the identical signature: the agent job's `Execute GitHub Copilot CLI` step gets a `400 Bad Request` from the Copilot API on its first turn (`failureClass=http_400_response_error`), after only ~1–1.4s, with zero tokens consumed and no output produced. The harness correctly classifies this as non-retryable and exits, so `SelfCare` never reaches the dispatcher step and no orchestrator work (dispatch or reliability review) happens for this package during the entire window.

**Action:** A gh-aw / Copilot CLI harness maintainer should investigate why the very first Copilot CLI turn for the `self-care` package's `agent` job returns HTTP 400 (likely a malformed/oversized prompt, unsupported request field, or model/config mismatch introduced by a recent change to `self-care.md` or its shared prompt template) and confirm a fix by re-running `self-care.lock.yml` and observing a completed `agent` job with `tokenCount > 0`.

Representative evidence

- 7/7 SelfCare scheduled runs in the window failed with the same signature:
- https://github.com/githubnext/gh-aw-cao/actions/runs/35044456568 (01:31:58Z)
- https://github.com/githubnext/gh-aw-cao/actions/runs/35042044893 (00:56:13Z)
- https://github.com/githubnext/gh-aw-cao/actions/runs/35037600716 (23:52:49Z)
- https://github.com/githubnext/gh-aw-cao/actions/runs/35036213504 (23:33:42Z)
- https://github.com/githubnext/gh-aw-cao/actions/runs/35035152179 (23:19:41Z)
- https://github.com/githubnext/gh-aw-cao/actions/runs/35033048641 (22:52:12Z)
- https://github.com/githubnext/gh-aw-cao/actions/runs/35031625813 (22:34:18Z)
- Common log excerpt from `agent/30_Execute GitHub Copilot CLI.txt` (identical across runs, only IDs/timestamps differ):
```
[copilot-harness] attempt 1: process started (pid=...)
400 Bad Request
Changes +0 -0
Duration 0s
Resume copilot --resume=
[copilot-harness] attempt 1 failed: exitCode=1 failureClass=http_400_response_error ... isHTTP400ResponseError=true ... tokenCount=0
[copilot-harness] attempt 1: HTTP 400 response error — not retrying (persistent request validation/state failure)
##[error]Agent execution exited with code 1
```
- Downstream `conclusion` job in each run correctly records "Handle agent failure" and "Report failed jobs", so the control plane fails closed and reports the run failure — but no dispatcher telemetry is ever emitted because the `agent` job never gets to `Emit control-plane dispatcher telemetry`.
- No other package in the 24h window shows this HTTP 400 signature; the ~500 workflow runs sampled contained only isolated single-occurrence failures elsewhere (e.g. `Deployed dashboard integration` cache-deletion 404s, `CAO Dashboard`/`Documentation Pages`/`Dashboard views` failures tied to fork-PR `action_required` gating or non-CAO test/build issues on `copilot/*` branches), none of which indicate a shared admission, dispatch, routing, or cache defect.

Collection notes

- The shared activity cache (`$RUNNER_TEMP/cao-activity/gh-aw-logs.sqlite`) was not populated in this run (cache miss), so evidence was gathered via bounded read-only GitHub Actions API queries (workflow runs list, job list, and job logs) scoped to `githubnext/gh-aw-cao`, covering the requested 24-hour window (2026-09-15T01:43:08Z snapshot start extended to the full prior 24h ending 2026-09-16T01:43:08Z; actual failures cluster in the last ~3 hours of that window, 22:30–01:32 UTC).

Agent prompt

Investigate why the Copilot CLI returns an immediate `400 Bad Request` (`http_400_response_error`, `tokenCount=0`) on the very first turn of the `self-care` orchestrator's `agent` job in `githubnext/gh-aw-cao`. Steps:
1. Inspect `.github/workflows/self-care.md` and any shared prompt/include files it composes (e.g. under `.github/workflows/shared/`) for recent changes that could produce a malformed or oversized prompt payload, or a bad model/tool configuration passed to the Copilot CLI.
2. Compare against other orchestrator workflows in the same repo that are succeeding, to isolate what is unique to `self-care`.
3. Reproduce locally if possible (e.g. `gh aw compile` then run the harness against the generated prompt) to see the exact 400 response body if available in `/tmp/gh-aw/sandbox/agent/logs/`.
4. Fix the root cause (e.g. prompt size/formatting, invalid config field) and validate with `npm run compile` plus a manual `workflow_dispatch` of `self-care.yml`, confirming the `agent` job completes with `tokenCount > 0` and no `http_400_response_error`.

Validation commands: `npm run typecheck:cao && npm test` (control-plane sources), `npm run compile` (regenerate `self-care.lock.yml` if the `.md` source changes).

**References:**
- https://github.com/githubnext/gh-aw-cao/actions/runs/35044456568
- https://github.com/githubnext/gh-aw-cao/actions/runs/35042044893
- https://github.com/githubnext/gh-aw-cao/actions/runs/35037600716

### Control Plane
- Correlation ID: 35044423050-118
- Central repository: githubnext/gh-aw-cao
- Control plane run: https://github.com/githubnext/gh-aw-cao/actions/runs/35044423050

> Generated by [CAO Evolution / Reliability](https://github.com/githubnext/gh-aw-cao/actions/runs/35044835701) · copilot · auto · 167.1 AIC · ⌖ 7.28 AIC · ⊞ 13.2K · [◷](https://github.com/search?q=repo%3Agithubnext%2Fgh-aw-cao+is%3Aissue+%22gh-aw-workflow-call-id%3A+githubnext%2Fgh-aw-cao%2Fcao-evolution-reliability%22&type=issues)
> - [x] expires on Sep 30, 2026, 1:52 AM UTC

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/self-care.md, its shared includes, and the generated self-care.lock.yml; compare them with succeeding orchestrator workflows. Reproduce with gh aw compile and inspect /tmp/gh-aw/sandbox/agent/logs/ for the HTTP 400 response, then run npm run compile and the stated typecheck and test commands. Done means a manual self-care workflow dispatch completes the agent job with tokenCount > 0 and no http_400_response_error.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, javascript
Domain
ci-cd, devops, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.