Make add-arm-resource Vally eval runnable (seed fixture + output graders)
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 144
Description
## Background
The `add-arm-resource` eval ([tools/azsdk-cli/Azure.Sdk.Tools.Vally/evals/tools/add-arm-resource.eval.yaml](https://github.com/Azure/azure-sdk-tools/blob/main/tools/azsdk-cli/Azure.Sdk.Tools.Vally/evals/tools/add-arm-resource.eval.yaml)) is currently a **stub**. It prompts the agent to author an ARM resource into a TypeSpec project, but:
- No project files are seeded on disk — the prompt names a project (`Microsoft.Widget`) that does not exist, so the agent has no real workspace to anchor on. This causes flaky skill invocation (raised by @haolingdong-msft in #15811).
- The only grader is `tool-calls` (requires `azsdk_typespec_generate_authoring_plan`). The flaky `edit`-tool assertion was removed; the real "did it author the resource" check (file output) is not yet in place.
## Tasks
- [ ] Seed a minimal starter TypeSpec project fixture under `Azure.Sdk.Tools.Vally/fixtures/` and copy it into the agent workdir via `environment.files` (mirrors the `azure-typespec-author` benchmark approach).
- [ ] Align the project name with the `Contoso.WidgetManager` convention used by the rest of the suite, instead of the one-off `Microsoft.Widget` name in the prompt.
- [ ] Add `file-exists` / `file-contains` graders on the produced `asset.tsp`.
- [ ] Add a `run-command` grader to verify `npx tsp compile` succeeds.
- [ ] Once real, wire it into a gated run.
## Context
- PR: #15811
- Reference setup: `.github/skills/azure-typespec-author/evaluate/` (#15694)
- Reviewer note: this Vally project is not yet wired into any pipeline (the skill-eval pipeline only runs `.github/skills` evals), so the stub does not block #15811.
Contributor guide
Research direction
Start with evals/tools/add-arm-resource.eval.yaml and the reference setup under .github/skills/azure-typespec-author/evaluate/. Add the starter fixture under Azure.Sdk.Tools.Vally/fixtures/ and configure environment.files for the agent workdir, then add output graders for asset.tsp and a run-command grader for npx tsp compile. Done means the eval uses the Contoso.WidgetManager project, verifies the authored file and content, and compiles successfully.
Written by the indexing model from the issue text.
Assessment
- Domain
- testing, tooling
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100