microsoft / microsoft/vscode-azureresourcegroups

Scaffold agent omits `allowedHosts: true` from vite.config, stalling the frontend preview approval gate

Open
#1,724 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
78
Forks
55
Avg merge
16h 49m
Merged PRs (30d)
139

Description

Summary

The scaffold agent produced a working full-stack project and then shipped a vite.config.ts without allowedHosts: true — the one setting the product's own instructions say is required for the frontend preview webview to load.

This was caught by the MSBench eval suite on its first run against real agent output (scaffold-fullstack, run 2026082620153444, 6/7, resolved: false). The defect was not present in any certification fixture.

Why this is a product bug and not an over-strict gate

I verified the gate's claim against the product's own rails documents before filing. The requirement is stated four times, with the consequence spelled out:

resources/agents/azure-project-scaffold/references/frontend-preview-steps.md:32

In vite.config set server: { host: true, allowedHosts: true, strictPort: false }allowedHosts: true stops the dev server 403-blocking the forwarded/webview origin

resources/agents/azure-project-scaffold.agent.md:53

the frontend you scaffolded MUST be embeddable in that iframe: its vite.config server sets host: true + allowedHosts: true + strictPort: falseIf the preview can't load, the user cannot click Approve UI and the whole flow stalls.

Also at resources/agents/azure-project-scaffold/instructions.md:173 and resources/agents/shared-references/architecture.md:391.

So this is not a case of the eval inventing a requirement. The agent violated an explicitly documented, repeatedly stated contract, and the documented consequence is a hard stall of the guided flow: the preview webview fails to load, the user cannot click Approve UI, and the scaffold→integrate hand-off never happens.

Evidence

FAIL: scaffolded frontend is preview-embeddable and keeps the API seam
  • [devServerRejectsWebviewOrigin] services/web/vite.config.ts:
    vite.config server must set `allowedHosts: true` or the dev server
    403-blocks the webview origin.

Run 2026082620153444 · scaffold-fullstack · completed 14m34s · exit 1.

Attribution checked against all four known traps before believing the red:

Trap Status
NOT_APPLICABLE / coverageGap marker on stderr absent
exit 3 (grader itself broke) no — exit 1
RATE_LIMIT / -> 429 no error.json at all
timeoutMs kill no — whole run 14m34s

Everything else in the run passed, which is what makes the red meaningful rather than noise:

  • validate-project-builds --require-frontendpassed. The project genuinely installs and builds.
  • validate-integration-plan --has-frontend — passed.
  • Frontend preview approval gate opened; no premature hand-off to integrate; no chat-question fallback; liveness sentinel passed.

So the agent did almost everything right, including the expensive part, and missed one line of config that stalls the user-visible flow.

Suggested direction

Worth considering why an explicitly and repeatedly documented requirement was missed, rather than only fixing the instance:

  • The requirement lives in a reference document (frontend-preview-steps.md) that the agent may not read on every path, while instructions.md:173 mentions it inside a long checklist item.
  • A generated vite.config.ts is a small, high-leverage artifact — a template or a post-scaffold check would enforce this more reliably than prose. The product already knows the exact required shape: server: { host: true, allowedHosts: true, strictPort: false }.
  • The same section lists sibling requirements with identical stall consequences (dev script must actually serve; no X-Frame-Options / frame-ancestors CSP; must not start a competing dev server). If prose is not sufficient for allowedHosts, it is likely not sufficient for those either — and the eval gate covers them, so we will find out.

Provenance

Found by the scaffold-fullstack eval stimulus on feat/CoR. This is the first defect the suite has found in real product output rather than in a fixture — the gate that caught it had been mutation-tested but never before run against a live agent.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with resources/agents/azure-project-scaffold.agent.md, instructions.md, and references/frontend-preview-steps.md, then inspect the generated services/web/vite.config.ts. Run the scaffold-fullstack evaluation or its preview-embeddability gate, using run 2026082620153444 as the failing case. Done means the generated frontend passes the allowedHosts check and the preview webview reaches the Approve UI gate.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript, vite
Domain
frontend, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.