[skills-eval] dotnet-blazor: 9 skills, 36% pass — 1 P0, 1 P1, 5 to strengthen, 2 keep

> ## 🔄 EDITED 2026-07-19 — correction per #909
>
> **This finding has been revised.** The errored trials that drove the **FIX-RELIABILITY (P0)** flag(s) below were shown in [#909](https://github.com/dotnet/skills/issues/909) to be **judge-side infrastructure failures** — a disabled/throttled PAT (`CAPIError 400 organization disabled`) and Vally's `session.idle` judge timeout — **not** fixture nondeterminism. Verdict scores already exclude errored trials, so re-running the classifier with those judge errors dropped reclassifies the affected skill(s):
>
> | Skill | ~~Was~~ | Now |
> | --- | --- | --- |
> | `configure-auth` | ~~FIX-RELIABILITY (P0)~~ | **EFFICIENT-WIN (keep)** |
>
> Struck-through text below is the **superseded** finding — in particular the "pin fixture SDK/tool versions" remediation does **not** apply. The title's P0 count has been updated. No eval re-run is required to correct the scores; recovering the lost judgments (optional) means **re-judging** those trials, not re-running the skill.

## Context — cross-family skill evaluation: `dotnet-blazor`

This issue is **self-contained**: it captures everything a skill author needs to act on the `dotnet-blazor` plugin without opening the full report.

**What this measures.** Every runnable skill in `dotnet/skills` was run through [Vally](https://github.com/microsoft/evaluate) (0.7) on a **cross-family matrix**: 5 executor model families — `opus-4.8`, `gpt-5.5`, `sonnet-4.6`, `haiku-4.5`, `mai-flash` — each judged by a **different family** (judge ≠ executor; default judge = latest Opus, or GPT when Opus is the executor). For every skill, a **skilled** run is compared against a **baseline** (no-skill) run and scored per executor. This removes single-model and self-judging bias, so a skill that only helps one model family — or only its own family's judge — is visible.

- **Data source:** cross-family CI grid (run `29228914412` + backfills) — 5 executors × 85 runnable skills, **419 scored cells** over 84 skills.
- **Row grain:** one row per **skill**, aggregated across its (up to 5) executor cells. `avgN` is the mean trial count behind the cells (trials 1–17; higher = more statistically trustworthy). `thin-N` flags directional-only rows.

### `dotnet-blazor` at a glance (portfolio scorecard)

| Plugin | Skills | Cells | Pass | Impact | Tie-trials | Err | avg ΔTok | avg ΔTurns | avg ΔTools | Headline |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| **dotnet-blazor** | 9 | 45 | 36% | 0.368 | 18 | 1 | +42,425 | +0.78 | +1.53 | Costly (`plan-ui-change` +408k); mixed |

**Insights.** 9 skill(s); mean impact **0.37**; 3/9 help ≥1 frontier model. No frontier-validated exemplar. Weak-model-only wins (frontier misses): `use-js-interop`. Weakest: `support-prerendering` (0.12).

~~**Address first:** `configure-auth` — reliability; `support-prerendering` — regression; `plan-ui-change` — cost.~~

**Address first:** `support-prerendering` — regression; `plan-ui-change` — cost. _(`configure-auth` reclassified out of P0 — see EDIT above.)_

### How to read the table

Each skill is scored **skilled vs. baseline** on these axes:

| Signal | Column | What it means | Good |
| --- | --- | --- | --- |
| Breadth | `Families ✓` (n/5) | How many of the **5 model families** the skill helps (per-family pass count, max `5/5`) | 4–5 / 5 |
| Where | `Passed on` | *Which* families passed. **Frontier = latest Opus + latest GPT** are **bold**; Sonnet 4.6 / Haiku 4.5 / MAI Flash are mid/low-weight | frontier ✓ |
| Magnitude | `Impact` (−1…+1) | How strongly the judge prefers *skilled* over *baseline* | ≥ 0.4 |
| Decisiveness | `Ties▵` | Trials where the judge saw **no difference** → skill is *inert* | low |
| Safety | `Loss▵` | Trials where **skilled was WORSE** than baseline → skill misfires | ~0 |
| Reliability | `Err` | Trials that errored/crashed in setup or judging | 0 |
| Efficiency | `ΔTok` / `ΔTurns` / `ΔTools` | Extra tokens / agent turns / tool calls vs baseline | ≤ 0 |
| Confidence | `avgN` | Mean trials behind the verdict; low N = directional only | ≥ 3 |
| Invocation | `Call%` | Share of skilled trials where the model actually **invoked the skill** | ~100% |

> **`Families ✓` is cell-level (max `5/5`); `Ties▵`/`Loss▵` are trial-level tallies summed across *all* families** (including the ones where the skill failed). A high `Families ✓` next to non-zero `Loss▵` is not a contradiction — see `Passed on` and the Action text for where losses landed.

**Action buckets** (each skill has one primary action; `[flags]` note secondary concerns):

| Bucket | Priority | Meaning |
| --- | --- | --- |
| FIX-RELIABILITY | 🔴 P0 | Errored trials / no verdict — stabilize the harness before trusting the score |
| FIX-DISCOVERY | 🔴 P0 | Model doesn't invoke it (`Call% < 50%`) — a triggering/description problem |
| FIX-REGRESSION | 🔴 P0 | Skilled is worse than baseline on many trials — the skill misfires |
| ADD-DECISIVENESS | 🟠 P1 | Called ~100% but ties dominate, ~0 impact — inert; needs sharper behavioral steps |
| TRIM-COST | 🟠 P1 | Passes but with heavy token/turn overhead — trim verbosity |
| EXEMPLAR | 🟢 keep | Broad, strong, reliable win — use as a template |
| EFFICIENT-WIN | 🟢 protect | Wins *and* cuts turns/tools — the ideal shape |
| KEEP-POLISH | 🟢 | Solid majority win; minor polish + more trials |
| STRENGTHEN | 🟡 P2 | Marginal/mixed lift — sharpen triggers & success criteria |

### Per-skill actions

| Skill | Families ✓ | Passed on (frontier **bold**) | Impact | Ties▵ | Loss▵ | Err | avgN | Call% | ΔTok | ΔTurns | ΔTools | Action |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `plan-ui-change` | 4/5 | **GPT**, Sonnet, Haiku, MAI | 0.54 | 1 | 1 | 0 | 5 | 100% | +407,812 | +8.68 | +10.24 | **TRIM-COST** · 🟠 **Passes but heavy (+407,812 tok).** Value is real; cost isn't justified. Trim: (1) cut redundant/whole-file reads — point to specific sections; (2) replace "explore everything" with a targeted checklist; (3) move deep reference material behind links instead of inlining it. *([frontier-miss], [costly])* |
| `coordinate-components` | 2/5 | **GPT**, Sonnet | 0.48 | 2 | 1 | 0 | 2 | 100% | +121,041 | +3.3 | +3.7 | **STRENGTHEN** · 🟡 **Marginal/mixed** (passes 2/5 — **GPT**, Sonnet). Diagnosis: mostly *ties* — too generic/non-prescriptive. Try: (1) sharpen the trigger so it fires only where it wins; (2) add 1–2 opinionated, concrete steps that change behaviour; (3) add trials to separate signal from noise. *([frontier-miss], [frontier-regressed], [costly])* |
| `create-blazor-project` | 0/5 | — | 0.45 | 0 | 2 | 0 | 3 | 100% | -392,992 | -6.93 | -4.47 | **STRENGTHEN** · 🟡 **Marginal/mixed** (passes 0/5). Diagnosis: misses both frontier models — likely assumes context they solve unaided. Try: (1) sharpen the trigger so it fires only where it wins; (2) add 1–2 opinionated, concrete steps that change behaviour; (3) add trials to separate signal from noise. If frontier models never benefit, scope it explicitly to weaker models or reconsider its value. *([both-frontier-miss])* |
| `configure-auth` | 2/5 | Haiku, MAI | 0.44 | 3 | 1 | 1 | 1.8 | 100% | +92,263 | -0.9 | -1.3 | ~~**FIX-RELIABILITY** · 🔴 **1 errored trial(s)** — verdict can't be trusted until setup is deterministic. This is an infra/harness fix, not a content one: (1) capture the failing trial's stderr; (2) pin tool/SDK versions in the fixture; (3) add a setup smoke-check before scoring. *([reliability], [both-frontier-miss], [frontier-regressed], [efficient])*~~

🔄 **EDIT (see #909):** **EFFICIENT-WIN** · ✅ Better *and* cheaper — wins on Haiku, MAI while saving 0.9 turns / 1.3 tools (weak-model win; both frontier models miss/regress, so don't template blindly). The 1 errored trial was a **judge-side `session.idle` timeout** (haiku), not fixture nondeterminism. *([both-frontier-miss], [frontier-regressed], [efficient])* |
| `author-component` | 2/5 | **GPT**, MAI | 0.43 | 2 | 0 | 0 | 3.8 | 100% | +78,872 | +3.46 | +4.39 | **STRENGTHEN** · 🟡 **Marginal/mixed** (passes 2/5 — **GPT**, MAI). Diagnosis: mostly *ties* — too generic/non-prescriptive. Try: (1) sharpen the trigger so it fires only where it wins; (2) add 1–2 opinionated, concrete steps that change behaviour; (3) add trials to separate signal from noise. *([frontier-miss])* |
| `use-js-interop` | 2/5 | Sonnet, Haiku | 0.38 | 2 | 2 | 0 | 3.8 | 85% | -20,440 | -1.17 | -1.57 | **EFFICIENT-WIN** · ✅ **Better *and* cheaper.** Wins on Sonnet, Haiku while saving 1.17 turns, 1.57 tools, 20,440 tok. What's good: it adds value without bloat — the ideal shape. Protect the brevity; don't let it grow. ⚠️ **But both frontier models miss** — this may only be helping mid/low-weight models. Treat as a weak-model win, not a universal one; check the frontier trajectories before templating. *([both-frontier-miss], [efficient])* |
| `collect-user-input` | 2/5 | Sonnet, MAI | 0.24 | 2 | 1 | 0 | 2 | 90% | +55,008 | +0.5 | +1.4 | **STRENGTHEN** · 🟡 **Marginal/mixed** (passes 2/5 — Sonnet, MAI). Diagnosis: mostly *ties* — too generic/non-prescriptive. Try: (1) sharpen the trigger so it fires only where it wins; (2) add 1–2 opinionated, concrete steps that change behaviour; (3) add trials to separate signal from noise. If frontier models never benefit, scope it explicitly to weaker models or reconsider its value. *([both-frontier-miss], [frontier-regressed])* |
| `fetch-and-send-data` | 1/5 | Sonnet | 0.22 | 4 | 1 | 0 | 2 | 70% | +87,601 | +3.7 | +5.7 | **STRENGTHEN** · 🟡 **Marginal/mixed** (passes 1/5 — Sonnet). Diagnosis: mostly *ties* — too generic/non-prescriptive. Try: (1) sharpen the trigger so it fires only where it wins; (2) add 1–2 opinionated, concrete steps that change behaviour; (3) add trials to separate signal from noise. If frontier models never benefit, scope it explicitly to weaker models or reconsider its value. *([both-frontier-miss], [frontier-regressed])* |
| `support-prerendering` | 1/5 | Haiku | 0.12 | 2 | 4 | 0 | 2 | 100% | -47,342 | -3.6 | -4.3 | **FIX-REGRESSION** · 🔴 **Regresses ~40% of trials** (worse than baseline), losing on GPT, Sonnet. The skill is over-applying. Fixes: (1) add explicit **stop-conditions** ("do NOT act when…"); (2) narrow the trigger to the exact scenario it helps; (3) demote prescriptive edits to *suggestions the agent can decline*. Prioritise the frontier miss (Opus, GPT). *([both-frontier-miss], [frontier-regressed])* |

---

_{Generated from the cross-family Call-to-Action report (`CALL-TO-ACTION.md` §4–§5; companion `IMPACT-ANALYSIS.md`). Regenerate the underlying tables with `node deep-metrics.mjs "$env:TEMP\cf-ci" agg-ci` → `node gen-cta-tables.mjs agg-ci`. Numbers are directional where `avgN` is low; treat single-trial cells as hypotheses to confirm with more runs.}