Azure / Azure/azure-sdk-tools

[Epic][TypeSpec Authoring][Benchmark] Continuous improvement

Open
#15,726 0 comments 0 reactions 1 assignee Claimed by @haolingdong-msft View on GitHub
AzSDK Tools Agent dev inner loop Epic
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
144

Description

Epic to track continuous improvement work for the TypeSpec authoring benchmark — ongoing investments after the initial Vally adoption (#14667) is done.

Scope (non-exhaustive, will grow over time):

- Continuously add and refine benchmark cases (basic & advanced dataplane, basic & advanced ARM, regression cases for real-world issues).
- Improve grader quality and judge prompts to better reflect TypeSpec authoring correctness.
- Reduce flakiness and total running time of the benchmark suite.
- Strengthen reporting and dashboards (per-run timing, pass-rate trends, per-category breakdowns).
- Track and adopt upstream Vally features/fixes that improve our benchmark experience.
- Tune skill prompts / instructions based on benchmark findings.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.