[Epic][TypeSpec Authoring][Benchmark] Continuous improvement
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 144
Description
Epic to track continuous improvement work for the TypeSpec authoring benchmark — ongoing investments after the initial Vally adoption (#14667) is done.
Scope (non-exhaustive, will grow over time):
- Continuously add and refine benchmark cases (basic & advanced dataplane, basic & advanced ARM, regression cases for real-world issues).
- Improve grader quality and judge prompts to better reflect TypeSpec authoring correctness.
- Reduce flakiness and total running time of the benchmark suite.
- Strengthen reporting and dashboards (per-run timing, pass-rate trends, per-category breakdowns).
- Track and adopt upstream Vally features/fixes that improve our benchmark experience.
- Tune skill prompts / instructions based on benchmark findings.
Contributor guide
Assessment
This issue has not been assessed yet.