Azure / Azure/azure-sdk-tools

Add generate-sdk-locally trigger eval and verify skill pipeline

Open
#16,361 0 comments 0 reactions 0 assignees View on GitHub
EngSys
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 2h
Merged PRs (30d)
130

Description

## Context

PR #15376 completed the Waza-to-Vally migration tracked by #15147. One narrow cleanup item remains: `azsdk-common-generate-sdk-locally` has its capability eval but is the only originally migrated skill without a routing eval.

Broader eval-coverage enforcement remains tracked by #16345.

## Tasks

- Add `.github/skills/azsdk-common-generate-sdk-locally/evals/trigger.eval.yaml`.
- Include representative trigger and anti-trigger prompts.
- Follow the existing `skill-invocation.required` and `skill-invocation.disallowed` conventions.
- Confirm the eval configuration passes Vally lint.
- Link one Azure DevOps skill-eval run where the affected skill shard completes with no failed evals.

## Acceptance criteria

- [ ] Trigger and anti-trigger routing coverage exists for `azsdk-common-generate-sdk-locally`.
- [ ] Vally lint succeeds.
- [ ] A linked Azure DevOps run shows the affected skill evals passing. A report-only or `continueOnError` aggregate success with failed tests underneath is not sufficient.

## Out of scope

- General skill/MCP coverage enforcement tracked by #16345.
- Plugin-tree Waza assets.
- Unrelated eval-quality work.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing capability eval for .github/skills/azsdk-common-generate-sdk-locally and the skill-invocation.required and skill-invocation.disallowed conventions. Add trigger and anti-trigger coverage in evals/trigger.eval.yaml, run Vally lint, and link an Azure DevOps skill-eval run showing the affected shard passing without failed evals.

Written by the indexing model from the issue text.

Assessment

Tech stack
yaml
Domain
testing, tooling
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.