Improve analyzing-dotnet-performance cross-family quality signal
- Dominant language
- C#
- Stars
- 5.4k
- Forks
- 415
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 81
Description
## Context
Follow-up to #885. The reliability investigation separated infrastructure failures from skill quality. Once transient comparison errors and unmatched trajectories are reported correctly, the cross-family evaluation still shows quality gaps that should be addressed independently.
## Evidence
From the cross-family evaluation:
- The skill passed 3 of 5 model families.
- Aggregate outcomes included 7 ties and 4 losses.
- The Opus family missed skill invocation in one cell despite 96% invocation overall.
- Tool-call overhead increased by 6.33 calls on average.
- Existing scenarios 8 and 9 have moderate overfitting risk because parts of their grading criteria are too closely tied to specific techniques or vocabulary.
## Proposed work
- Rewrite scenarios 8 and 9 toward outcome-focused, independently evaluable rubrics without weakening the intended performance findings.
- Analyze tie and loss cases across model families to identify missing guidance or ambiguous fixtures.
- Investigate the Opus activation miss without broadening the skill beyond its documented scope.
- Reduce unnecessary tool usage while preserving diagnosis quality.
- Keep `eval.yaml` and `eval.vally.yaml` semantically aligned for any test changes.
## Acceptance criteria
- Static validation passes for `plugins/dotnet-diag`.
- Updated rubrics are classified as outcome-focused rather than technique/vocabulary-focused.
- Cross-family evaluation shows no regression in currently passing families.
- Ties/losses and tool-call overhead improve, with any remaining model-specific gaps documented.
- Reliability infrastructure changes remain scoped to #885 and are not mixed into this work.
Contributor guide
Assessment
This issue has not been assessed yet.