[TypeSpec Authoring][Eval] Prevent forced ARM cases from stopping for clarification
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 143
Description
## Goal
Update the forced ARM eval test cases so the agent has enough information to complete the task autonomously and does not stop to ask clarification questions during testing.
Parent epic: #15652
## Pipeline evidence
- Pipeline run: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6696207
- Failed ARM-template task: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6696207&view=logs&j=a9a86f66-3acd-5d69-e100-cec815754788&t=9508ee8f-d364-546c-18fe-560242d931a8
- Model: `gpt-5.6-sol`
- Forced ARM-template result: 8/11 cases passed
## Analysis
“Agentic search” below means `web_fetch`; “authoring plan” means `azsdk_typespec_generate_authoring_plan`.
| Case | Result | Routing | Validation MCP | Finding |
| --- | --- | --- | --- | --- |
| 002001 change resource type | Pass | Agentic search | Yes | — |
| 002002 add extension resource | Pass | Agentic search | Yes | — |
| 002003 define full update operation | **Fail** | Agentic search | **No** | Agent stopped to ask for confirmation instead of applying the PATCH operation. No edit was made, so `ArmCustomPatchSync` was absent. |
| 002004 extension resource from ProxyResource | Pass | Agentic search | Yes | — |
| 002005 define resource | Pass | Authoring plan | Yes | — |
| 002006 define child resource | **Fail** | Agentic search | **No** | Agent asked which API version should introduce the child resource and stopped. No resource definition was added. |
| 002007 define custom action | Pass | Both | Yes | — |
| 002008 add parameters | Pass | Both | Yes | — |
| 002009 add PATCH operation | **Fail** | Authoring plan | Yes | Agent produced valid `ArmResourcePatchSync`, while the grader required `ArmCustomPatchSync<...>`. This is a separate expectation mismatch, not a clarification-stop failure. |
| 002010 synchronous action | Pass | Agentic search | Yes | — |
| 002011 check-existence operation | Pass | Authoring plan | Yes | — |
The clarification-stop failures are concentrated in:
- `.github/skills/azure-typespec-author/evaluate/evals/002003.eval.yaml`
- `.github/skills/azure-typespec-author/evaluate/evals/002006.eval.yaml`
## Proposed work
- Make the 002003 forced prompt explicitly authorize applying the requested update operation without waiting for confirmation.
- Make the 002006 forced prompt specify the API version and any other required resource decisions so no clarification is needed.
- Review the forced-case prompt pattern and state that the supplied requirements are complete and the agent should proceed autonomously.
- Keep the intended routing for these covered cases: agentic search rather than the authoring-plan MCP.
## Acceptance criteria
- 002003 and 002006 complete without asking the user a question or ending with a clarification request.
- Both cases make the expected TypeSpec edits and call `azsdk_run_typespec_validation`.
- Both cases pass their forced-mode graders in a rerun of `azure-typespec-author-benchmark`.
- The updated prompts remain realistic while explicitly resolving decisions that would otherwise require user input.
- The unrelated 002009 PATCH-template expectation mismatch is tracked/fixed separately or explicitly excluded from this issue.
Contributor guide
Assessment
This issue has not been assessed yet.