Agentic perf skills: polish + retire select-agent-models and agentic-perf-reviewer agent
- Dominant language
- C#
- Stars
- 5.4k
- Forks
- 415
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 81
Description
## Summary
Proposing a consolidated polish + retirement PR across the `dotnet-ai`
perf-skills cluster, dogfooded against a reference MAF + Aspire + Foundry
app over the last several weeks.
## Scope
### Retirements
- **Retire `agentic-perf-reviewer.agent.md`.** The agent's responsibilities
collapse cleanly into `scan-agentic-app-perf` (audit) + `configure-agentic-perf-rules`
(always-on rules). An always-loaded agent was duplicating skill content
with no extra leverage and risked drifting from its constituent skills.
- **Retire the `select-agent-models` skill.** Its substance (per-agent
model selection guidance) now lives as a rule in
`configure-agentic-perf-rules` (rule #3) and a check in
`scan-agentic-app-perf` (`model.same-default`). The standalone skill
was rarely invoked directly and added a routing layer with no payoff.
### Rename
- **`audit-agentic-app-perf` → `scan-agentic-app-perf`** to match how
users describe the task ("scan my agent for perf issues"), which
improves intent matching per the CONTRIBUTING.md naming guidance.
### Per-skill changes
- **`configure-agentic-perf-rules` v0.3.0**
- Drop arbitrary numeric ceilings from rules #1 (agent count) and #2
(handoff edges). Reframed as "no hard ceiling — justify the add"
based on user feedback that the ceilings were producing
unjustified-by-data warnings.
- Idempotent v0.2.0 → v0.3.0 upgrade test added to `eval.yaml`.
- **`scan-agentic-app-perf`**
- Replace cryptic codes (`MA1`, `MA4`, etc.) with category slugs
(`model.same-default`, `apphost.model-literal-only`) — easier to
grep, easier to cite.
- Collapse multi-file output into a single, overwritten `scan.md`
plus a stable `latest-scan.md` alias.
- Pruned threshold-based checks where the threshold was guesswork;
kept the ones with evidence.
- **`setup-maf-evals` v2 spec rewrite**
- MSTest project shape replacing the old console-runner default;
matches the upstream
[Learn doc](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/evaluate-with-reporting)
and [ai-samples](https://github.com/dotnet/ai-samples/blob/main/src/microsoft-extensions-ai-evaluation/api/)
reference. Console shape still available behind `--shape console`.
- Three evaluator tiers: **NLP** (deterministic), **Quality**
(LLM-as-judge), **Safety** (Foundry).
- Auto-install of the `aieval` dotnet tool; per-run
`metrics-glossary.md`; HTML report via
`Microsoft.Extensions.AI.Evaluation.Reporting`.
- Rubric-driven `RubricEvaluator` template (custom criteria without
rolling your own evaluator).
- **Bug fix:** snake_case JSON loader sketches (`InputsLoader`,
`MatrixLoader`, etc.) now show
`JsonNamingPolicy.SnakeCaseLower` + `PropertyNameCaseInsensitive`
options. The original templates relied on STJ defaults, causing
PascalCase properties to bind to `null` (loud in Quality mode,
silently corrupting in Telemetry mode).
- **Deprecation note:** flag `Azure.AI.Inference` beta SDK (the
foundation for `AddAzureChatCompletionsClient`) as retiring on
August 26, 2026 per the
[Foundry supported-languages doc](https://learn.microsoft.com/en-us/azure/foundry-classic/foundry-models/supported-languages).
Add `AddOpenAIClient(...).AddChatClient(...)` to the
detection-pattern table for the OpenAI/v1 + stable OpenAI SDK path.
Update the reasoning-model `max_tokens` pitfall to recommend the
OpenAI SDK migration as the durable fix (related: dotnet/extensions#7580).
- **Cross-cutting**
- Topology-agnostic descriptions: drop "MAF + Aspire + Foundry"
framing in favor of "MAF; Aspire/Foundry optional". Add
"Supported topologies" body sections noting that plain console,
ASP.NET Core, and worker service shapes are all supported, with
AppHost-specific checks silently skipped when no AppHost is
detected. (No detection-logic changes in this PR — pure doc.
Deeper non-Aspire support is a candidate follow-up.)
- Harmonize `WHEN` / `NOT-WHEN` phrasing across all three skills.
- `CODEOWNERS`: rotate the perf-skills second owner; delete the
dangling entries for the retired `select-agent-models` skill and
`agentic-perf-reviewer.agent.md`.
- Common-pitfalls + check-id glossary docs added across the three
skills.
## Validation
- Dogfooded against a reference MAF + Aspire + Foundry app
(`code-review-buddy`) — `scan` produces 0 critical / 0 warn / 2 info
(both intentional-by-design), `setup-maf-evals` scaffolds and all
three modes (telemetry, quality, compare) produce clean reports.
- Skill-validator `eval.yaml` tests updated for the rule reframing
and the v0.2.0 → v0.3.0 upgrade path.
- `configure-agentic-perf-rules` description char count: 966 / 1024.
- `scan-agentic-app-perf` description char count: 994 / 1024.
- `setup-maf-evals` description char count: 1001 / 1024.
## PR
Will be opened as a draft against `dotnet/skills:main` from
`leslierichardson95:lerich/agentic-app-perf-skills`. Commit history is
intentionally fine-grained (each commit reflects a real iteration); can
be squashed on request before merge.
## Out of scope (deferred to follow-ups)
- Deeper non-Aspire detection refactor — current PR is doc-only on the
topology question.
- `mcp-csharp-*` CODEOWNERS rotation (separate concern).
- Updating the fork's `main` — will sync after this PR merges.
Contributor guide
Assessment
This issue has not been assessed yet.