Azure / Azure/azure-sdk-tools

[TypeSpec Authoring] Assess assessment skill agent friendliness

Open
#16,998 0 comments 0 reactions 0 assignees View on GitHub
AzSDK Tools Agent dev inner loop needs-triage
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 2h
Merged PRs (30d)
130

Description

## Parent issue

Part of #14529.

## Goal

Assess and improve the TypeSpec assessment skill's agent friendliness so agents can discover, invoke, interpret, and act on it reliably.

## Scope

- Define agent-friendliness criteria for skill routing, instructions, tool usage, outputs, errors, and mitigation handoffs.
- Evaluate whether an agent can select the correct assessment mode and supply required inputs without undocumented assumptions.
- Identify ambiguous, duplicated, or overly long instructions that cause inconsistent agent behavior.
- Assess whether reports and structured outputs make findings, evidence, incomplete analysis, and next actions unambiguous to an agent.
- Create prioritized improvements and regression cases for identified gaps.

## Acceptance criteria

- Agent-friendliness criteria and representative scenarios are documented.
- The initial skill is assessed against each criterion with evidence.
- Findings distinguish routing, instruction, execution, output-contract, and recovery issues.
- Each confirmed gap has a linked improvement or benchmark task.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named in the issue. Start with the TypeSpec assessment skill and parent issue #14529, then define assessment criteria and representative scenarios before evaluating routing, instructions, execution, outputs, and recovery. Done means documented evidence, prioritized improvements or benchmark tasks, and a linked improvement for every confirmed gap.

Written by the indexing model from the issue text.

Assessment

Domain
ai-infra-agents, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.