githubnext / githubnext/ado-aw
debug: Noop smoke pipeline fails to pull non-existent ghcr.io/github/gh-aw-mcpg:v0.3.7 (manifest unknown)
- Dominant language
- Rust
- Stars
- 23
- Forks
- 8
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 22
Description
## Summary
The `Daily safe-output smoke noop` pipeline (Azure DevOps definition `Noop smoke`, id `2506`, in `msazuresphere/AgentPlayground`) is failing in the **Agent** job at the `Pre-pull AWF and MCPG container images (v0.25.44)` step. The compiled lock file generated by ado-aw v0.30.1 (and also v0.30.2 on `main`) pins the MCP Gateway image to `ghcr.io/github/gh-aw-mcpg:v0.3.7`, **a tag that does not exist in the registry**. Docker returns `Error response from daemon: manifest unknown` and the job exits.
This has caused every daily smoke run since at least 16 May to fail.
## Run under investigation
- Pipeline: **Noop smoke** (definition id `2506`)
- Build: **607153** — `Daily safe-output smoke noop-607153`
- URL: https://dev.azure.com/msazuresphere/AgentPlayground/_build/results?buildId=607153&view=logs&j=6c181f5c-fa64-5337-60e0-f5bfb300ad55&t=7a6f47d6-c3f0-5ed7-7b53-50831ee9d6a7
- Branch / commit: `refs/heads/main` @ `9597f35c` (source repo: `githubnext/ado-aw`)
- Source pipeline file: `tests/safe-outputs/noop.lock.yml`
- Compiler version recorded in lock header: **`@ado-aw source="tests/safe-outputs/noop.md" version=0.30.1`**
- Compiled with embedded AWF version: **`v0.25.44`**
- Started: 2026-05-18 13:48:08 UTC — Finished: 2026-05-18 13:50:07 UTC (≈2 min)
- Result: **failed**
Recent run history (definition 2506) shows uninterrupted failures:
| Build | Result | Started (UTC) |
|--------|----------|----------------------|
| 607153 | failed | 2026-05-18 13:48:08 |
| 607067 | failed | 2026-05-18 03:36:54 |
| 606890 | failed | 2026-05-17 03:36:53 |
| 606819 | failed | 2026-05-16 21:27:24 |
| 606813 | failed | 2026-05-16 20:28:59 |
| 606812 | failed | 2026-05-16 20:14:19 |
## Failure classification
| Stage / Job | Step | Result |
|-------------|-----------------------------------------------------|--------|
| `__default` Stage | — | failed |
| `Agent` Phase / Job | — | failed |
| **Pre-pull AWF and MCPG container images (v0.25.44)** (order 16, log 19) | Bash exited with code 1 | **failed (root cause)** |
Category per debug rubric: **Agent — download/setup → AWF Container Startup**, specifically an MCPG image pull failure caused by the **compiler emitting a non-existent image tag** in the generated YAML.
## Root cause
`tests/safe-outputs/noop.lock.yml` contains this step (verbatim from the generated file on `main` and on `v0.30.2`):
```bash
docker pull ghcr.io/github/gh-aw-firewall/squid:0.25.44
docker pull ghcr.io/github/gh-aw-firewall/agent:0.25.44
docker tag ghcr.io/github/gh-aw-firewall/squid:0.25.44 ghcr.io/github/gh-aw-firewall/squid:latest
docker tag ghcr.io/github/gh-aw-firewall/agent:0.25.44 ghcr.io/github/gh-aw-firewall/agent:latest
docker pull ghcr.io/github/gh-aw-mcpg:v0.3.7
```
The squid (`0.25.44`) and agent (`0.25.44`) images pull successfully. The final pull of `ghcr.io/github/gh-aw-mcpg:v0.3.7` fails:
```
2026-05-18T13:49:43.1698657Z ghcr.io/github/gh-aw-firewall/agent:0.25.44
2026-05-18T13:49:43.3763131Z Error response from daemon: manifest unknown
2026-05-18T13:49:43.3838247Z ##[error]Bash exited with code '1'.
```
Querying `ghcr.io/v2/github/gh-aw-mcpg` directly (anonymous pull token):
- `GET /v2/github/gh-aw-mcpg/manifests/v0.3.7` → `404 Not Found`
- `GET /v2/github/gh-aw-mcpg/manifests/0.3.7` → `404 Not Found`
The published tag list for `github/gh-aw-mcpg` jumps from `v0.2.16` directly to nothing higher — there is no `v0.3.x` series. Tags currently present include `latest`, `v0.0.94` … `v0.2.16` and matching commit-SHA tags. **No `v0.3.7` was ever pushed.**
The lock.yml on `main` (compiler v0.30.1) and on tag `v0.30.2` (released 2026-05-18 13:33 UTC, ~15 min before this build queued) both still emit `ghcr.io/github/gh-aw-mcpg:v0.3.7`, so upgrading the compiler to the latest release does **not** resolve this.
This is a compiler bug: ado-aw is generating a `docker pull` line that pins MCPG to a tag that does not exist in the published registry. Either the compiler's default MCPG tag constant needs to be moved back to a published tag (e.g. `v0.2.16`) or the missing `gh-aw-mcpg:v0.3.7` image needs to be published.
## Evidence
Tail of `Pre-pull AWF and MCPG container images (v0.25.44)` log:
```
2026-05-18T13:49:18.6247280Z Digest: sha256:bc13486f73cc9b8327e276839f936ecf2492cf86a9bee9114d2b018bdfc54782
2026-05-18T13:49:18.6264430Z Status: Downloaded newer image for ghcr.io/github/gh-aw-firewall/squid:0.25.44
...
2026-05-18T13:49:43.1679145Z Digest: sha256:71511dd944cd13a4d980a57bcf7a071f8662cb482686c58db294d9903e0960e4
2026-05-18T13:49:43.1691265Z Status: Downloaded newer image for ghcr.io/github/gh-aw-firewall/agent:0.25.44
2026-05-18T13:49:43.1698657Z ghcr.io/github/gh-aw-firewall/agent:0.25.44
2026-05-18T13:49:43.3763131Z Error response from daemon: manifest unknown
2026-05-18T13:49:43.3838247Z ##[error]Bash exited with code '1'.
2026-05-18T13:49:43.3838247Z ##[section]Finishing: Pre-pull AWF and MCPG container images (v0.25.44)
```
Timeline issue record:
```json
{
"type": "error",
"category": "General",
"message": "Bash exited with code '1'.",
"data": { "source": "TaskInternal", "logFileLineNumber": "141" }
}
```
ghcr.io registry probe (anonymous pull token):
```
GET https://ghcr.io/v2/github/gh-aw-mcpg/manifests/v0.3.7 → 404 Not Found
GET https://ghcr.io/v2/github/gh-aw-mcpg/manifests/0.3.7 → 404 Not Found
Highest tag in /tags/list: v0.2.16
```
## Classification
- **Category:** `bug` — the ado-aw compiler emits an incorrect generated YAML that references a non-existent container image tag.
- Not a user-configuration issue: the failing step and the version pin are entirely produced by the compiler, with no user-tunable input.
- Not transient: the same step has failed identically on every run since at least 16 May, and the same `v0.3.7` pin is still present on `main` and on the most recent release tag `v0.30.2`.
## Recommended next steps (for maintainers — not implemented as part of this report)
- Either publish `ghcr.io/github/gh-aw-mcpg:v0.3.7`, or downgrade the MCPG tag baked into the compiler default to a tag that exists (e.g. `v0.2.16`).
- Cut a new ado-aw release so downstream lock files can be re-compiled.
- Consider adding a release-time check that all referenced container image tags resolve before publishing.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.