githubnext / githubnext/ado-aw
debug: v0.31.1 compiler emits different YAML than committed *.lock.yml files from #861, breaking integrity check
- Dominant language
- Rust
- Stars
- 23
- Forks
- 8
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 22
Description
# debug: v0.31.1 compiler emits different YAML than committed `*.lock.yml` files generated by PR #861, breaking `ado-aw check`
## Summary
The scheduled `Daily safe-output smoke: noop` pipeline (ADO org `msazuresphere`, project `AgentPlayground`) fails in the **"Verify pipeline integrity"** step. The committed `tests/safe-outputs/noop.lock.yml` on `main` contains an `Emit aw_info.json` bash step that the publicly-released `ado-aw` v0.31.1 binary does **not** reproduce when re-compiling the same source. Both the on-disk lock and the regenerated lock claim `version=0.31.1` in their headers, so the integrity check cannot reconcile them.
The same drift very likely affects all 26 lock files touched by [#861 "regen ado-aw"](https://github.com/githubnext/ado-aw/pull/861) — `noop.lock.yml` is just the first one to be exercised by a scheduled smoke run.
## Failing build
- **URL:** https://dev.azure.com/msazuresphere/AgentPlayground/_build/results?buildId=611238&view=logs&j=6c181f5c-fa64-5337-60e0-f5bfb300ad55&t=e16f045e-3223-5d9a-19b7-d1e5a64bc357
- **Build ID:** 611238
- **Definition:** `Noop smoke`
- **Branch:** `refs/heads/main` (commit fetched at build time)
- **Reason:** `manual` (re-run shortly after #861 merged)
- **Start / Finish:** 2026-06-05 13:31:37Z → 13:33:43Z
- **Failed stage / job / step:** `__default` → `Agent` → **`Verify pipeline integrity`** (log id 13)
- **Pool:** `AZS-1ES-L-Playground-ubuntu-22.04`
## Failing log excerpt (verbatim)
```
--- tests/safe-outputs/noop.lock.yml (on disk)
+++ tests/safe-outputs/noop.lock.yml (expected from source)
@@ -261,16 +261,6 @@
displayName: "ado-aw"
- bash: |
- set -eo pipefail
-
- mkdir -p "$(Agent.TempDirectory)/staging"
- cat >"$(Agent.TempDirectory)/staging/aw_info.json" <<'AW_INFO_EOF'
- {"agent_name":"Daily safe-output smoke: noop","build_definition_id":"$(System.DefinitionId)","build_id":"$(Build.BuildId)","compiler_version":"0.31.1","engine":"copilot","model":"gpt-5-mini","org":"","repo":"","schema":"ado-aw/aw_info/1","source":"tests/safe-outputs/noop.md","source_branch":"$(Build.SourceBranch)","source_version":"$(Build.SourceVersion)","target":"standalone"}
- AW_INFO_EOF
- displayName: "Emit aw_info.json"
- condition: always()
-
- - bash: |
cat >> "/tmp/awf-tools/agent-prompt.md" << 'SAFEOUTPUTS_EOF'
---
Summary: 0 line(s) added, 10 line(s) removed
Error: Integrity check failed: generated pipeline for 'Daily safe-output smoke: noop' does not match tests/safe-outputs/noop.lock.yml. Re-run `ado-aw compile` to update the pipeline file.
##[error]Bash exited with code '1'.
```
## What I verified
I reproduced this locally on a clean checkout to rule out environmental noise:
1. **Source on `main` is unchanged from what the build saw.** `tests/safe-outputs/noop.md` does not configure or reference anything that would conditionally enable / disable an `aw_info.json` emit.
2. **The publicly-released v0.31.1 compiler does *not* emit the `aw_info.json` step.**
```text
> .\ado-aw.exe --version
ado-aw 0.31.1
> .\ado-aw.exe compile tests/safe-outputs/noop.md
Generated standalone pipeline: tests/safe-outputs/noop.lock.yml
> Select-String -Path tests\safe-outputs\noop.lock.yml -Pattern 'aw_info|Emit'
(no matches)
```
3. **The committed `noop.lock.yml` on `main` *does* contain the step** (lines around 261–270 per the diff). Both files carry the same header:
```
# @ado-aw source="tests/safe-outputs/noop.md" version=0.31.1
```
4. **#861 introduced the step.** `git log` on `tests/safe-outputs/noop.lock.yml`:
```
ea16d851 2026-06-05T13:21:52Z regen ado-aw (#861)
9597f35c 2026-05-18T13:43:54Z chore: regenerate all ado-aw workflows with v0.30.1 (#619)
```
#861's patch on `noop.lock.yml` bumps the version pin `0.30.1 → 0.31.1` *and* adds the entire `Emit aw_info.json` block as a new step.
5. **The failing build started 10 minutes after #861 merged**, so the trigger is unambiguous: the regen committed YAML the integrity check (running the public release) refuses to accept.
6. **`--debug-pipeline` is not a flag on the released v0.31.1 binary.** I tested it; the binary suggests `--debug` (logging only) and rejects `--debug-pipeline`. So the drift cannot be explained by "regen ran with `--debug-pipeline`, integrity check ran without it" against the released artifact.
## Likely root cause
The compiler binary that produced PR #861's lock files is **not** the same binary the public `v0.31.1` GitHub release tag ships. It is most likely a build from a commit *after* the `v0.31.1` tag (which added the `aw_info.json` emit logic) but whose hard-coded version string is still `0.31.1`.
This is effectively a versioning hole:
- The lock-file header is the only thing `ado-aw check` uses to assert "this lock matches this compiler version."
- If two different codegen behaviors both stamp `version=0.31.1`, the integrity check has no way to distinguish them, and CI regressions caused by an unreleased-but-tag-named build of the compiler land silently in `main`.
## Impact
- **All 26 lock files touched by #861** are likely in the same drift state. The noop smoke just happens to be on a daily schedule, so it was the first to surface the regression.
- Every scheduled / manual run of those pipelines will fail at "Verify pipeline integrity" until either:
- the lock files are regenerated using whatever release artifact actually corresponds to the published v0.31.1 binary, **or**
- a new release (e.g. `v0.31.2`) is cut from the compiler revision that #861 used, and the lock files' headers are bumped to match.
## Suggested fixes (out of scope for this report; for maintainer triage)
1. **Bump the version pin whenever codegen changes.** Even patch-level codegen changes should bump the version in the lock-file header so `ado-aw check` reliably catches drift.
2. **Embed the build SHA in the lock header** (e.g. `version=0.31.1+sha.abcdef0`) so unreleased-but-tag-stamped builds are distinguishable.
3. **Gate the regen workflow on `--version` matching the latest published release**, so PRs like #861 can't be authored using a local build that disagrees with what the integrity check uses.
4. **Re-release** (cut a `v0.31.2` from the codegen revision that produced #861) and regenerate all lock files against that public binary.
## Environment
- ADO org / project: `msazuresphere` / `AgentPlayground`
- ADO pool: `AZS-1ES-L-Playground-ubuntu-22.04`
- Local reproduction: Windows 11, `ado-aw.exe 0.31.1` downloaded by our `scripts/rebuild-pipelines.ps1` from `https://github.com/githubnext/ado-aw/releases/download/v0.31.1/ado-aw-windows-x64.exe`
- Related upstream context: [#861](https://github.com/githubnext/ado-aw/pull/861) (the regen PR that introduced the drift)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.