microsoft / microsoft/vscode-azureresourcegroups

Deploy writes `scaffold-manifest.validationResult` in a shape the contract rejects

Open
#1,787 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
78
Forks
55
Avg merge
16h 49m
Merged PRs (30d)
139

Description

Found while running a real end-to-end Create New Project with Copilot project through to IaC generation. The generated infrastructure compiles cleanly, but the deploy agent wrote scaffold-manifest.json.validationResult in a shape the contract does not allow.

What it wrote

"validationResult": {
  "status": "passed",
  "runners": ["az bicep build", "manual conformance audit"]
}

What the contract requires

azure-deploy/scaffold/references/scaffold-schemas.ts:

export interface ValidationResult {
  status: "Validated" | "Partial" | "Failed";
  checks: ValidationCheck[];
  proof?: string;
}

"passed" is not a legal status, and runners is not checks.

iac-compiles caught it:

FAIL — the generated infrastructure compiles, but the scaffold manifest is wrong:
  • [iacNotValidated] status is "passed", expected "Validated"
  • [missingBicepBuildCheck] no check records compiling the template

Why it matters beyond the eval

azure-deploy/deploy/instructions.md:48 has the deploy sub-phase read validationResult as an input, and approval-gate-template.md:25 renders {scaffold-manifest.json.validationResult.status} straight into the approval gate the user sees. A malformed value degrades the deploy phase and the gate, not just the grader.

The self-report contradiction is the interesting part: the agent claims in runners that it ran az bicep build — and it evidently did, because the template genuinely compiles (verified independently: az bicep build --file infra/main.bicep --stdout → 0 errors, 1 linter advisory, ARM JSON emitted). It just recorded that fact in a shape nothing downstream can read.

The instruction is not the problem

Worth stating explicitly, because it rules out the obvious fix. validation-and-manifest.md § 11c is already about as strong as prose gets:

  • a literal JSON example with "status": "Validated" and "checks": [...]
  • Field names are fixed — copy them exactly, naming status + checks[] and { name, passed: boolean, detail? }
  • explicit anti-patterns: "passed is a boolean — not "result": "PASS", not "status"", "The array is checks — not notes, warnings, or errors"
  • an explicit warning not to substitute ConformanceResult

So this is agent non-compliance against a maximally explicit instruction. Making the prose louder is unlikely to help; a schema validation step on write, or a repair pass before the deploy gate reads it, seems more promising.

Also found in the same run (minor)

infra/modules/resources.bicep:209 hardcodes a cloud-specific hostname:

var kvUrl = 'https://${keyVaultName}.vault.azure.net/secrets'

Bicep's own linter flags this (no-hardcoded-env-urls); it breaks sovereign clouds where the suffix differs. environment() is the portable form. iac-compiles reports it as an advisory rather than a failure, which seems like the right calibration.

Repro notes

Prompt: "Build a task tracker with a React frontend, an Azure Functions HTTP backend, and a PostgreSQL database for durable storage. Uploaded attachments are held in Blob Storage." — stopped at the deploy gate; no Azure resources were provisioned. n=1.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with azure-deploy/scaffold/references/scaffold-schemas.ts and the failing iac-compiles output; trace how validationResult is written and consumed by azure-deploy/deploy/instructions.md:48 and approval-gate-template.md:25. Done means generated manifests use the ValidationResult status/checks shape and record the compile check without breaking the deploy gate. Treat infra/modules/resources.bicep:209 as a separate advisory.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, typescript
Domain
devops, infrastructure, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.