Azure / Azure/azure-sdk-tools

[Epic][TypeSpec Authoring][Assessment Skill] Implement Assessment Skill

Open
#14,529 0 comments 0 reactions 1 assignee Claimed by @haolingdong-msft View on GitHub
AzSDK Tools Agent design epic dev inner loop Epic needs-triage
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
143

Description

Our approach focuses on facilitating TypeSpec authoring. Upon initialization, the agent server conducts an analysis of the user project opened within the current context, gathering essential project metadata. Once this process is complete, the server awaits user prompts to initiate the execution of the first scenario. Each TypeSpec authoring scenario is independent. We need an assessment scenario to tie everything together and create an "assess → propose → update" loop for TypeSpec authoring assistance. The purpose of this issue is to clarify what is required for the assessment.

**What exactly we want the "assessment" step to evaluate?**

To enable a continuous "assess → propose → update" loop within the TypeSpec authoring assistant, the assessment stage must analyze the current state of the user's TypeSpec project and identify actionable insights. The focus areas include:
- Static Code Quality - as long as anything not covered by linter, but with documented guidance, or pre-defined skills (like .NET naming convention checks)
- API Specification Completeness - missing request/response models; properties without types; error model gaps, etc.
- Adoption of Recommended Patterns - does it use long-running patterns correctly? Is pagination, filtering, versioning applied when applicable?

**Assessment Output Format**

The assistant should output:
- A ranked list of issues sorted by estimated impact: Mandatory - blocks code generation; inconsistencies; missing decorators; incomplete responses. Optional - missing description; style, naming, or optimization suggestions.
- Suggested next action per issue
- Optional: A completeness score - API modeling: 70%; Code Quality: 85%; Pattern Compliance: 60%. This can help motivate and guide iterative improvement.

**Implementation Principles for the Assessment Loop**

To make the loop smooth and reliable:
1. Project-wide scan on activation, build a full interpreted form of the user project.
2. Lightweight incremental re-assessment, trigger after major edits or assistant-applied updates.
3. Semantic-aware diffing, detect structural change, not just textual edits.
4. Propose actions only when confident, high-certainty issues get auto-fixes, lower certainty ask the user.
5. Intent-aware assessment, align checks with user-stated goals ("I'm building a new resource...")

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.