Azure / Azure/azure-sdk-tools

Make add-arm-resource Vally eval runnable (seed fixture + output graders)

Open
#16,024 0 comments 0 reactions 0 assignees View on GitHub
azsdk-cli
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
144

Description

## Background

The `add-arm-resource` eval ([tools/azsdk-cli/Azure.Sdk.Tools.Vally/evals/tools/add-arm-resource.eval.yaml](https://github.com/Azure/azure-sdk-tools/blob/main/tools/azsdk-cli/Azure.Sdk.Tools.Vally/evals/tools/add-arm-resource.eval.yaml)) is currently a **stub**. It prompts the agent to author an ARM resource into a TypeSpec project, but:

- No project files are seeded on disk — the prompt names a project (`Microsoft.Widget`) that does not exist, so the agent has no real workspace to anchor on. This causes flaky skill invocation (raised by @haolingdong-msft in #15811).
- The only grader is `tool-calls` (requires `azsdk_typespec_generate_authoring_plan`). The flaky `edit`-tool assertion was removed; the real "did it author the resource" check (file output) is not yet in place.

## Tasks

- [ ] Seed a minimal starter TypeSpec project fixture under `Azure.Sdk.Tools.Vally/fixtures/` and copy it into the agent workdir via `environment.files` (mirrors the `azure-typespec-author` benchmark approach).
- [ ] Align the project name with the `Contoso.WidgetManager` convention used by the rest of the suite, instead of the one-off `Microsoft.Widget` name in the prompt.
- [ ] Add `file-exists` / `file-contains` graders on the produced `asset.tsp`.
- [ ] Add a `run-command` grader to verify `npx tsp compile` succeeds.
- [ ] Once real, wire it into a gated run.

## Context

- PR: #15811
- Reference setup: `.github/skills/azure-typespec-author/evaluate/` (#15694)
- Reviewer note: this Vally project is not yet wired into any pipeline (the skill-eval pipeline only runs `.github/skills` evals), so the stub does not block #15811.

Contributor guide

Open the contributing guide

Research direction

Start with evals/tools/add-arm-resource.eval.yaml and the reference setup under .github/skills/azure-typespec-author/evaluate/. Add the starter fixture under Azure.Sdk.Tools.Vally/fixtures/ and configure environment.files for the agent workdir, then add output graders for asset.tsp and a run-command grader for npx tsp compile. Done means the eval uses the Contoso.WidgetManager project, verifies the authored file and content, and compiles successfully.

Written by the indexing model from the issue text.

Assessment

Domain
testing, tooling
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.