microsoft / microsoft/vscode-azureresourcegroups
Deploy writes `scaffold-manifest.validationResult` in a shape the contract rejects
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 78
- Forks
- 55
- Avg merge
- 16h 49m
- Merged PRs (30d)
- 139
Description
Found while running a real end-to-end Create New Project with Copilot project through to IaC generation. The generated infrastructure compiles cleanly, but the deploy agent wrote scaffold-manifest.json.validationResult in a shape the contract does not allow.
What it wrote
"validationResult": {
"status": "passed",
"runners": ["az bicep build", "manual conformance audit"]
}
What the contract requires
azure-deploy/scaffold/references/scaffold-schemas.ts:
export interface ValidationResult {
status: "Validated" | "Partial" | "Failed";
checks: ValidationCheck[];
proof?: string;
}
"passed" is not a legal status, and runners is not checks.
iac-compiles caught it:
FAIL — the generated infrastructure compiles, but the scaffold manifest is wrong:
• [iacNotValidated] status is "passed", expected "Validated"
• [missingBicepBuildCheck] no check records compiling the template
Why it matters beyond the eval
azure-deploy/deploy/instructions.md:48 has the deploy sub-phase read validationResult as an input, and approval-gate-template.md:25 renders {scaffold-manifest.json.validationResult.status} straight into the approval gate the user sees. A malformed value degrades the deploy phase and the gate, not just the grader.
The self-report contradiction is the interesting part: the agent claims in runners that it ran az bicep build — and it evidently did, because the template genuinely compiles (verified independently: az bicep build --file infra/main.bicep --stdout → 0 errors, 1 linter advisory, ARM JSON emitted). It just recorded that fact in a shape nothing downstream can read.
The instruction is not the problem
Worth stating explicitly, because it rules out the obvious fix. validation-and-manifest.md § 11c is already about as strong as prose gets:
- a literal JSON example with
"status": "Validated"and"checks": [...] - ⛔ Field names are fixed — copy them exactly, naming
status+checks[]and{ name, passed: boolean, detail? } - explicit anti-patterns: "
passedis a boolean — not"result": "PASS", not"status"", "The array ischecks— notnotes,warnings, orerrors" - an explicit warning not to substitute
ConformanceResult
So this is agent non-compliance against a maximally explicit instruction. Making the prose louder is unlikely to help; a schema validation step on write, or a repair pass before the deploy gate reads it, seems more promising.
Also found in the same run (minor)
infra/modules/resources.bicep:209 hardcodes a cloud-specific hostname:
var kvUrl = 'https://${keyVaultName}.vault.azure.net/secrets'
Bicep's own linter flags this (no-hardcoded-env-urls); it breaks sovereign clouds where the suffix differs. environment() is the portable form. iac-compiles reports it as an advisory rather than a failure, which seems like the right calibration.
Repro notes
Prompt: "Build a task tracker with a React frontend, an Azure Functions HTTP backend, and a PostgreSQL database for durable storage. Uploaded attachments are held in Blob Storage." — stopped at the deploy gate; no Azure resources were provisioned. n=1.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with azure-deploy/scaffold/references/scaffold-schemas.ts and the failing iac-compiles output; trace how validationResult is written and consumed by azure-deploy/deploy/instructions.md:48 and approval-gate-template.md:25. Done means generated manifests use the ValidationResult status/checks shape and record the compile check without breaking the deploy gate. Treat infra/modules/resources.bicep:209 as a separate advisory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, typescript
- Domain
- devops, infrastructure, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100