Azure / Azure/azure-sdk-tools

[TypeSpec Authoring] Support fast assessment mode

Open
#16,811 0 comments 0 reactions 0 assignees Claimed by @haolingdong-msft View on GitHub
AzSDK Tools Agent dev inner loop needs-triage
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 2h
Merged PRs (30d)
130

Description

## Problem

A complete TypeSpec assessment provides detailed semantic intent and impact analysis, but service developers also need a faster feedback path while iterating. The assessment skill should expose an explicit fast mode that focuses on actionable risk without producing a separate or lower-quality interpretation of REST, downstream, or compliance findings.

This supports the assess → propose → update loop tracked by #14529.

## Goal

Add an impact-only fast assessment mode that reduces turnaround time by omitting semantic intent generation while preserving evidence-backed REST breaking changes, REST-compatible downstream breaking changes, and Azure compliance findings.

## Fast-mode contract

**Fast mode = full assessment − semantic intents/diffs.**

- Keep REST breaking-change assessment.
- Keep SDK/downstream breaking-change assessment.
- Keep documentation-grounded Azure compliance assessment.
- Keep overall code-safety and confidence indicators.
- Keep exact changed TypeSpec source evidence and actionable source links.
- Omit semantic intent generation, semantic metrics, and non-actionable change explanations.
- Use the same finding structure, severity model, terminology, and visual design as complete mode.
- Do not report “no impact” when analysis is incomplete or blocked.

## Requirements

- Let the caller explicitly select `complete` or `fast`; default behavior must remain documented and predictable.
- Produce a standalone `fast-assessment.html` report.
- Use bounded model input and require an explicit approve/reject judgment for every deterministic REST and downstream candidate.
- Derive generated-client impact from confirmed REST changes when no separate downstream candidate exists.
- Detect important TypeSpec-only downstream changes, including paging metadata, client location/flattening changes, and LRO-to-synchronous operation changes.
- Require every approved finding to include actual behavior, expected behavior, evidence, affected operations where applicable, and exact changed TypeSpec source.
- Record preparation, judgment, rendering, and total execution time so fast-mode performance can be compared with complete mode.
- Preserve deterministic behavior and fail clearly when required evidence is unavailable.

## Acceptance criteria

- A user can request fast mode and receive `fast-assessment.html` without semantic intent content or semantic metrics.
- For the same change set, fast and complete reports agree on REST, downstream, compliance, severity, and overall safety conclusions.
- Every actionable fast-mode finding links to the corresponding changed TypeSpec source.
- Fast mode renders the same finding UX as complete mode, excluding only semantic sections.
- Regression coverage includes known REST-breaking, downstream-only, compliance-failure, no-impact, and blocked-analysis cases.
- The curated historical PR set is reassessed in both modes, with finding parity and execution times recorded.
- Documentation explains mode selection, output location, evidence requirements, and limitations.

Contributor guide

Open the contributing guide

Research direction

Use the existing complete assessment as the baseline for adding explicit complete and fast mode selection, then define the standalone fast-assessment.html output and timing records. Cover the listed REST, downstream, compliance, no-impact, and blocked-analysis cases, and reassess the curated historical PR set in both modes. Done means fast reports omit semantic content while preserving finding parity, evidence links, safety conclusions, and documented limitations.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.