githubnext / githubnext/ado-aw

debug: Noop smoke pipeline fails to pull non-existent ghcr.io/github/gh-aw-mcpg:v0.3.7 (manifest unknown)

Open
#621 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
23
Forks
8
Avg merge
4d 9h
Merged PRs (30d)
22

Description

## Summary

The `Daily safe-output smoke noop` pipeline (Azure DevOps definition `Noop smoke`, id `2506`, in `msazuresphere/AgentPlayground`) is failing in the **Agent** job at the `Pre-pull AWF and MCPG container images (v0.25.44)` step. The compiled lock file generated by ado-aw v0.30.1 (and also v0.30.2 on `main`) pins the MCP Gateway image to `ghcr.io/github/gh-aw-mcpg:v0.3.7`, **a tag that does not exist in the registry**. Docker returns `Error response from daemon: manifest unknown` and the job exits.

This has caused every daily smoke run since at least 16 May to fail.

## Run under investigation

- Pipeline: **Noop smoke** (definition id `2506`)
- Build: **607153** — `Daily safe-output smoke noop-607153`
- URL: https://dev.azure.com/msazuresphere/AgentPlayground/_build/results?buildId=607153&view=logs&j=6c181f5c-fa64-5337-60e0-f5bfb300ad55&t=7a6f47d6-c3f0-5ed7-7b53-50831ee9d6a7
- Branch / commit: `refs/heads/main` @ `9597f35c` (source repo: `githubnext/ado-aw`)
- Source pipeline file: `tests/safe-outputs/noop.lock.yml`
- Compiler version recorded in lock header: **`@ado-aw source="tests/safe-outputs/noop.md" version=0.30.1`**
- Compiled with embedded AWF version: **`v0.25.44`**
- Started: 2026-05-18 13:48:08 UTC — Finished: 2026-05-18 13:50:07 UTC (≈2 min)
- Result: **failed**

Recent run history (definition 2506) shows uninterrupted failures:

| Build | Result | Started (UTC) |
|--------|----------|----------------------|
| 607153 | failed | 2026-05-18 13:48:08 |
| 607067 | failed | 2026-05-18 03:36:54 |
| 606890 | failed | 2026-05-17 03:36:53 |
| 606819 | failed | 2026-05-16 21:27:24 |
| 606813 | failed | 2026-05-16 20:28:59 |
| 606812 | failed | 2026-05-16 20:14:19 |

## Failure classification

| Stage / Job | Step | Result |
|-------------|-----------------------------------------------------|--------|
| `__default` Stage | — | failed |
| `Agent` Phase / Job | — | failed |
| **Pre-pull AWF and MCPG container images (v0.25.44)** (order 16, log 19) | Bash exited with code 1 | **failed (root cause)** |

Category per debug rubric: **Agent — download/setup → AWF Container Startup**, specifically an MCPG image pull failure caused by the **compiler emitting a non-existent image tag** in the generated YAML.

## Root cause

`tests/safe-outputs/noop.lock.yml` contains this step (verbatim from the generated file on `main` and on `v0.30.2`):

```bash
docker pull ghcr.io/github/gh-aw-firewall/squid:0.25.44
docker pull ghcr.io/github/gh-aw-firewall/agent:0.25.44
docker tag ghcr.io/github/gh-aw-firewall/squid:0.25.44 ghcr.io/github/gh-aw-firewall/squid:latest
docker tag ghcr.io/github/gh-aw-firewall/agent:0.25.44 ghcr.io/github/gh-aw-firewall/agent:latest
docker pull ghcr.io/github/gh-aw-mcpg:v0.3.7
```

The squid (`0.25.44`) and agent (`0.25.44`) images pull successfully. The final pull of `ghcr.io/github/gh-aw-mcpg:v0.3.7` fails:

```
2026-05-18T13:49:43.1698657Z ghcr.io/github/gh-aw-firewall/agent:0.25.44
2026-05-18T13:49:43.3763131Z Error response from daemon: manifest unknown
2026-05-18T13:49:43.3838247Z ##[error]Bash exited with code '1'.
```

Querying `ghcr.io/v2/github/gh-aw-mcpg` directly (anonymous pull token):

- `GET /v2/github/gh-aw-mcpg/manifests/v0.3.7` → `404 Not Found`
- `GET /v2/github/gh-aw-mcpg/manifests/0.3.7` → `404 Not Found`

The published tag list for `github/gh-aw-mcpg` jumps from `v0.2.16` directly to nothing higher — there is no `v0.3.x` series. Tags currently present include `latest`, `v0.0.94` … `v0.2.16` and matching commit-SHA tags. **No `v0.3.7` was ever pushed.**

The lock.yml on `main` (compiler v0.30.1) and on tag `v0.30.2` (released 2026-05-18 13:33 UTC, ~15 min before this build queued) both still emit `ghcr.io/github/gh-aw-mcpg:v0.3.7`, so upgrading the compiler to the latest release does **not** resolve this.

This is a compiler bug: ado-aw is generating a `docker pull` line that pins MCPG to a tag that does not exist in the published registry. Either the compiler's default MCPG tag constant needs to be moved back to a published tag (e.g. `v0.2.16`) or the missing `gh-aw-mcpg:v0.3.7` image needs to be published.

## Evidence

Tail of `Pre-pull AWF and MCPG container images (v0.25.44)` log:

```
2026-05-18T13:49:18.6247280Z Digest: sha256:bc13486f73cc9b8327e276839f936ecf2492cf86a9bee9114d2b018bdfc54782
2026-05-18T13:49:18.6264430Z Status: Downloaded newer image for ghcr.io/github/gh-aw-firewall/squid:0.25.44
...
2026-05-18T13:49:43.1679145Z Digest: sha256:71511dd944cd13a4d980a57bcf7a071f8662cb482686c58db294d9903e0960e4
2026-05-18T13:49:43.1691265Z Status: Downloaded newer image for ghcr.io/github/gh-aw-firewall/agent:0.25.44
2026-05-18T13:49:43.1698657Z ghcr.io/github/gh-aw-firewall/agent:0.25.44
2026-05-18T13:49:43.3763131Z Error response from daemon: manifest unknown
2026-05-18T13:49:43.3838247Z ##[error]Bash exited with code '1'.
2026-05-18T13:49:43.3838247Z ##[section]Finishing: Pre-pull AWF and MCPG container images (v0.25.44)
```

Timeline issue record:

```json
{
"type": "error",
"category": "General",
"message": "Bash exited with code '1'.",
"data": { "source": "TaskInternal", "logFileLineNumber": "141" }
}
```

ghcr.io registry probe (anonymous pull token):

```
GET https://ghcr.io/v2/github/gh-aw-mcpg/manifests/v0.3.7 → 404 Not Found
GET https://ghcr.io/v2/github/gh-aw-mcpg/manifests/0.3.7 → 404 Not Found
Highest tag in /tags/list: v0.2.16
```

## Classification

- **Category:** `bug` — the ado-aw compiler emits an incorrect generated YAML that references a non-existent container image tag.
- Not a user-configuration issue: the failing step and the version pin are entirely produced by the compiler, with no user-tunable input.
- Not transient: the same step has failed identically on every run since at least 16 May, and the same `v0.3.7` pin is still present on `main` and on the most recent release tag `v0.30.2`.

## Recommended next steps (for maintainers — not implemented as part of this report)

- Either publish `ghcr.io/github/gh-aw-mcpg:v0.3.7`, or downgrade the MCPG tag baked into the compiler default to a tag that exists (e.g. `v0.2.16`).
- Cut a new ado-aw release so downstream lock files can be re-compiled.
- Consider adding a release-time check that all referenced container image tags resolve before publishing.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.