dotnet / dotnet/skills

Improve analyzing-dotnet-performance cross-family quality signal

Open
#886 1 comment 0 reactions 1 assignee Claimed by @artl93 View on GitHub
area-dotnet-diag task Triaged
Dominant language
C#
Stars
5.4k
Forks
415
Avg merge
1d 5h
Merged PRs (30d)
81

Description

## Context

Follow-up to #885. The reliability investigation separated infrastructure failures from skill quality. Once transient comparison errors and unmatched trajectories are reported correctly, the cross-family evaluation still shows quality gaps that should be addressed independently.

## Evidence

From the cross-family evaluation:

- The skill passed 3 of 5 model families.
- Aggregate outcomes included 7 ties and 4 losses.
- The Opus family missed skill invocation in one cell despite 96% invocation overall.
- Tool-call overhead increased by 6.33 calls on average.
- Existing scenarios 8 and 9 have moderate overfitting risk because parts of their grading criteria are too closely tied to specific techniques or vocabulary.

## Proposed work

- Rewrite scenarios 8 and 9 toward outcome-focused, independently evaluable rubrics without weakening the intended performance findings.
- Analyze tie and loss cases across model families to identify missing guidance or ambiguous fixtures.
- Investigate the Opus activation miss without broadening the skill beyond its documented scope.
- Reduce unnecessary tool usage while preserving diagnosis quality.
- Keep `eval.yaml` and `eval.vally.yaml` semantically aligned for any test changes.

## Acceptance criteria

- Static validation passes for `plugins/dotnet-diag`.
- Updated rubrics are classified as outcome-focused rather than technique/vocabulary-focused.
- Cross-family evaluation shows no regression in currently passing families.
- Ties/losses and tool-call overhead improve, with any remaining model-specific gaps documented.
- Reliability infrastructure changes remain scoped to #885 and are not mixed into this work.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.