[skills-eval] dotnet-msbuild: 14 skills, 56% pass — 2 P0, 1 P1, 2 to strengthen, 9 keep

## Context — cross-family skill evaluation: `dotnet-msbuild`

This issue is **self-contained**: it captures everything a skill author needs to act on the `dotnet-msbuild` plugin without opening the full report.

**What this measures.** Every runnable skill in `dotnet/skills` was run through [Vally](https://github.com/microsoft/evaluate) (0.7) on a **cross-family matrix**: 5 executor model families — `opus-4.8`, `gpt-5.5`, `sonnet-4.6`, `haiku-4.5`, `mai-flash` — each judged by a **different family** (judge ≠ executor; default judge = latest Opus, or GPT when Opus is the executor). For every skill, a **skilled** run is compared against a **baseline** (no-skill) run and scored per executor. This removes single-model and self-judging bias, so a skill that only helps one model family — or only its own family's judge — is visible.

- **Data source:** cross-family CI grid (run `29228914412` + backfills) — 5 executors × 85 runnable skills, **419 scored cells** over 84 skills.
- **Row grain:** one row per **skill**, aggregated across its (up to 5) executor cells. `avgN` is the mean trial count behind the cells (trials 1–17; higher = more statistically trustworthy). `thin-N` flags directional-only rows.

### `dotnet-msbuild` at a glance (portfolio scorecard)

| Plugin | Skills | Cells | Pass | Impact | Tie-trials | Err | avg ΔTok | avg ΔTurns | avg ΔTools | Headline |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| **dotnet-msbuild** | 14 | 70 | 56% | 0.282 | 31 | 0 | +8,450 | +0.28 | +0.01 | Deepest, most reliable plugin; thin-N |

**Insights.** 14 skill(s); mean impact **0.28**; 9/14 help ≥1 frontier model. Exemplars (win incl. frontier): `msbuild-server`, `build-perf-baseline`, `build-perf-diagnostics`, `directory-build-organization`, `including-generated-files`. Weakest: `eval-performance` (-0.24).

**Address first:** `msbuild-antipatterns` — discovery; `eval-performance` — regression; `check-bin-obj-clash` — cost.

### How to read the table

Each skill is scored **skilled vs. baseline** on these axes:

| Signal | Column | What it means | Good |
| --- | --- | --- | --- |
| Breadth | `Families ✓` (n/5) | How many of the **5 model families** the skill helps (per-family pass count, max `5/5`) | 4–5 / 5 |
| Where | `Passed on` | *Which* families passed. **Frontier = latest Opus + latest GPT** are **bold**; Sonnet 4.6 / Haiku 4.5 / MAI Flash are mid/low-weight | frontier ✓ |
| Magnitude | `Impact` (−1…+1) | How strongly the judge prefers *skilled* over *baseline* | ≥ 0.4 |
| Decisiveness | `Ties▵` | Trials where the judge saw **no difference** → skill is *inert* | low |
| Safety | `Loss▵` | Trials where **skilled was WORSE** than baseline → skill misfires | ~0 |
| Reliability | `Err` | Trials that errored/crashed in setup or judging | 0 |
| Efficiency | `ΔTok` / `ΔTurns` / `ΔTools` | Extra tokens / agent turns / tool calls vs baseline | ≤ 0 |
| Confidence | `avgN` | Mean trials behind the verdict; low N = directional only | ≥ 3 |
| Invocation | `Call%` | Share of skilled trials where the model actually **invoked the skill** | ~100% |

> **`Families ✓` is cell-level (max `5/5`); `Ties▵`/`Loss▵` are trial-level tallies summed across *all* families** (including the ones where the skill failed). A high `Families ✓` next to non-zero `Loss▵` is not a contradiction — see `Passed on` and the Action text for where losses landed.

**Action buckets** (each skill has one primary action; `[flags]` note secondary concerns):

| Bucket | Priority | Meaning |
| --- | --- | --- |
| FIX-RELIABILITY | 🔴 P0 | Errored trials / no verdict — stabilize the harness before trusting the score |
| FIX-DISCOVERY | 🔴 P0 | Model doesn't invoke it (`Call% < 50%`) — a triggering/description problem |
| FIX-REGRESSION | 🔴 P0 | Skilled is worse than baseline on many trials — the skill misfires |
| ADD-DECISIVENESS | 🟠 P1 | Called ~100% but ties dominate, ~0 impact — inert; needs sharper behavioral steps |
| TRIM-COST | 🟠 P1 | Passes but with heavy token/turn overhead — trim verbosity |
| EXEMPLAR | 🟢 keep | Broad, strong, reliable win — use as a template |
| EFFICIENT-WIN | 🟢 protect | Wins *and* cuts turns/tools — the ideal shape |
| KEEP-POLISH | 🟢 | Solid majority win; minor polish + more trials |
| STRENGTHEN | 🟡 P2 | Marginal/mixed lift — sharpen triggers & success criteria |

### Per-skill actions

| Skill | Families ✓ | Passed on (frontier **bold**) | Impact | Ties▵ | Loss▵ | Err | avgN | Call% | ΔTok | ΔTurns | ΔTools | Action |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `binlog-generation` | 0/5 | — | 0.64 | 3 | 0 | 0 | 3 | 93% | +13,103 | +0.87 | +0.4 | **STRENGTHEN** · 🟡 **Marginal/mixed** (passes 0/5). Diagnosis: mostly *ties* — too generic/non-prescriptive. Try: (1) sharpen the trigger so it fires only where it wins; (2) add 1–2 opinionated, concrete steps that change behaviour; (3) add trials to separate signal from noise. If frontier models never benefit, scope it explicitly to weaker models or reconsider its value. *([both-frontier-miss])* |
| `msbuild-server` | 5/5 | **Opus**, **GPT**, Sonnet, Haiku, MAI | 0.52 | 0 | 0 | 0 | 1 | 100% | +12,929 | +1 | +0.2 | **EXEMPLAR** · ✅ **Template-worthy.** What's good: broad (passes 5/5, incl. frontier Opus, GPT); zero ties or losses (clean). Keep as-is; lift its structure (crisp triggers + imperative steps) into weaker siblings. *([thin-N])* |
| `build-perf-baseline` | 4/5 | **GPT**, Sonnet, Haiku, MAI | 0.44 | 1 | 0 | 0 | 1 | 100% | -161,315 | -4.4 | -5 | **EXEMPLAR** · ✅ **Template-worthy.** What's good: broad (passes 4/5, incl. frontier GPT); cheaper (4.4 turns, 5 tools, 161,315 tok). Keep as-is; lift its structure (crisp triggers + imperative steps) into weaker siblings. ⚠️ Caveat: frontier miss on Opus — confirm it isn't only lifting weaker models on cases frontier already handles. *([frontier-miss], [efficient], [thin-N])* |
| `build-perf-diagnostics` | 4/5 | **GPT**, Sonnet, Haiku, MAI | 0.44 | 1 | 0 | 0 | 1 | 80% | +30,394 | -1.6 | -0.6 | **EXEMPLAR** · ✅ **Template-worthy.** What's good: broad (passes 4/5, incl. frontier GPT); cheaper (1.6 turns, 0.6 tools). Keep as-is; lift its structure (crisp triggers + imperative steps) into weaker siblings. ⚠️ Caveat: frontier miss on Opus — confirm it isn't only lifting weaker models on cases frontier already handles. *([frontier-miss], [efficient], [thin-N])* |
| `directory-build-organization` | 5/5 | **Opus**, **GPT**, Sonnet, Haiku, MAI | 0.40 | 0 | 0 | 0 | 1 | 100% | -28,616 | -2 | -2.2 | **EXEMPLAR** · ✅ **Template-worthy.** What's good: broad (passes 5/5, incl. frontier Opus, GPT); zero ties or losses (clean); cheaper (2 turns, 2.2 tools, 28,616 tok). Keep as-is; lift its structure (crisp triggers + imperative steps) into weaker siblings. *([efficient], [thin-N])* |
| `including-generated-files` | 5/5 | **Opus**, **GPT**, Sonnet, Haiku, MAI | 0.40 | 0 | 0 | 0 | 1 | 100% | +28,420 | +1.4 | +1.4 | **EXEMPLAR** · ✅ **Template-worthy.** What's good: broad (passes 5/5, incl. frontier Opus, GPT); zero ties or losses (clean). Keep as-is; lift its structure (crisp triggers + imperative steps) into weaker siblings. *([thin-N])* |
| `build-parallelism` | 2/5 | Sonnet, Haiku | 0.28 | 3 | 0 | 0 | 1 | 100% | +21,857 | +1 | +2 | **STRENGTHEN** · 🟡 **Marginal/mixed** (passes 2/5 — Sonnet, Haiku). Diagnosis: mostly *ties* — too generic/non-prescriptive. Try: (1) sharpen the trigger so it fires only where it wins; (2) add 1–2 opinionated, concrete steps that change behaviour; (3) add trials to separate signal from noise. If frontier models never benefit, scope it explicitly to weaker models or reconsider its value. *([both-frontier-miss], [thin-N])* |
| `resolve-project-references` | 3/5 | **Opus**, Sonnet, Haiku | 0.28 | 1 | 1 | 0 | 1 | 100% | +4,776 | +0.4 | -2 | **KEEP-POLISH** · 🟢 **Solid majority win** on **Opus**, Sonnet, Haiku. Losses/ties are concentrated in the **non-passing** cells (MAI, GPT), not the 3 passing ones — the passes are clean. To firm up: (1) raise trials on GPT, MAI; (2) investigate why a frontier model (GPT) didn't pass — that's the highest-value gap; (3) minor wording polish only. *([frontier-miss], [thin-N])* |
| `binlog-failure-analysis` | 3/5 | **GPT**, Haiku, MAI | 0.24 | 2 | 0 | 0 | 1 | 100% | -111,566 | -2.2 | -1.6 | **KEEP-POLISH** · 🟢 **Solid majority win** on **GPT**, Haiku, MAI. Losses/ties are concentrated in the **non-passing** cells (Opus, Sonnet), not the 3 passing ones — the passes are clean. To firm up: (1) raise trials on Opus, Sonnet; (2) investigate why a frontier model (Opus) didn't pass — that's the highest-value gap; (3) minor wording polish only. *([frontier-miss], [thin-N])* |
| `incremental-build` | 3/5 | **GPT**, Sonnet, MAI | 0.24 | 2 | 0 | 0 | 1 | 100% | +56,173 | +2.4 | +2.6 | **KEEP-POLISH** · 🟢 **Solid majority win** on **GPT**, Sonnet, MAI. Losses/ties are concentrated in the **non-passing** cells (Opus, Haiku), not the 3 passing ones — the passes are clean. To firm up: (1) raise trials on Opus, Haiku; (2) investigate why a frontier model (Opus) didn't pass — that's the highest-value gap; (3) minor wording polish only. *([frontier-miss], [thin-N])* |
| `check-bin-obj-clash` | 2/5 | **Opus**, **GPT** | 0.20 | 2 | 1 | 0 | 1 | 100% | +154,323 | +4.4 | +4.4 | **TRIM-COST** · 🟠 **Passes but heavy (+154,323 tok).** Value is real; cost isn't justified. Trim: (1) cut redundant/whole-file reads — point to specific sections; (2) replace "explore everything" with a targeted checklist; (3) move deep reference material behind links instead of inlining it. *([costly], [thin-N])* |
| `msbuild-modernization` | 3/5 | Sonnet, Haiku, MAI | 0.16 | 1 | 1 | 0 | 1 | 100% | +37,125 | +0 | -1.4 | **KEEP-POLISH** · 🟢 **Solid majority win** on Sonnet, Haiku, MAI. Losses/ties are concentrated in the **non-passing** cells (Opus, GPT), not the 3 passing ones — the passes are clean. To firm up: (1) raise trials on Opus, GPT; (2) investigate why a frontier model (Opus, GPT) didn't pass — that's the highest-value gap; (3) minor wording polish only. *([both-frontier-miss], [frontier-regressed], [thin-N])* |
| `msbuild-antipatterns` | 0/5 | — | -0.05 | 13 | 4 | 0 | 4 | 30% | +8,217 | -0.15 | +0.3 | **FIX-DISCOVERY** · 🔴 **Barely invoked (30% of trials)** — the model isn't calling it, so content can't help yet. Fix discovery first: (1) rewrite the `description:` with concrete trigger phrases a user would actually type; (2) add "USE WHEN…" bullets; (3) shorten the name/scope so it competes for the model's attention. Re-measure before editing the body. *([both-frontier-miss], [frontier-regressed], [not-invoked])* |
| `eval-performance` | 0/5 | — | -0.24 | 2 | 3 | 0 | 1 | 100% | +52,485 | +2.8 | +1.6 | **FIX-REGRESSION** · 🔴 **Regresses ~60% of trials** (worse than baseline), losing on Opus, GPT, Sonnet. The skill is over-applying. Fixes: (1) add explicit **stop-conditions** ("do NOT act when…"); (2) narrow the trigger to the exact scenario it helps; (3) demote prescriptive edits to *suggestions the agent can decline*. Prioritise the frontier miss (Opus, GPT). *([both-frontier-miss], [frontier-regressed], [thin-N])* |

### ⚠️ Coverage gap — 4 skills excluded from the cross-family run

The matrix scores a skill only if it has a `tests/dotnet-msbuild//eval.vally.yaml`. The following `dotnet-msbuild` skills ship a `SKILL.md` but were **never measured** — so they carry **zero** evidence of impact (not counted in the pass rate or scorecard above):

- **Has a base `eval.yaml` but no `eval.vally.yaml`** — add a cross-family Vally config to fold them into the matrix: `extension-points`, `item-management`, `property-patterns`, `target-authoring`

Until these have an `eval.vally.yaml`, we cannot tell whether they help, hurt, or are inert. **Recommended:** add the cross-family eval configs and re-run so they appear in the next scorecard.

---

_{Generated from the cross-family Call-to-Action report (`CALL-TO-ACTION.md` §4–§5; companion `IMPACT-ANALYSIS.md`). Regenerate the underlying tables with `node deep-metrics.mjs "$env:TEMP\cf-ci" agg-ci` → `node gen-cta-tables.mjs agg-ci`. Numbers are directional where `avgN` is low; treat single-trial cells as hypotheses to confirm with more runs.}