dotnet / dotnet/core

Release-notes agent: improve coverage so foundational features aren't missed (Preview 5 retrospective)

Open
#10,435 0 comments 0 reactions 0 assignees View on GitHub
needs-area-label
Dominant language
PowerShell
Stars
22k
Forks
4.9k
Avg merge
5d 5h
Merged PRs (30d)
29

Description

## Summary

The .NET 11 Preview 5 release-notes generation missed two foundational Blazor features that shipped under `area-blazor` + `feature-validation`, plus one borderline OpenAPI fix. The misses surfaced only when validating the draft by building working samples against the Preview 5 build. Filing this to capture concrete improvements to the release-notes skills so a future preview's draft catches them automatically.

## What was missed

- [dotnet/aspnetcore#66526](https://github.com/dotnet/aspnetcore/pull/66526) — Async form validation (`AddValidationTask`, `IsValidationPending`, `IsValidationFaulted`). No milestone tag.
- [dotnet/aspnetcore#66646](https://github.com/dotnet/aspnetcore/pull/66646) — Validation resource-based localization (`AddValidationLocalization()`). Labeled `area-infrastructure`, easy to skip in a Blazor-component slice.
- [dotnet/aspnetcore#65650](https://github.com/dotnet/aspnetcore/pull/65650) — MVC + OpenAPI multiple `Produces` per status code preserved. No milestone tag.

All three were present in `changes.json` with full metadata. The input data was fine; the editorial pipeline filtered them out.

## Root causes

1. **`generate-features` editorial-scoring stage was skipped.** Stub `features.json` was written and sub-agents got slim `{id, title, url, labels}` JSON, bypassing the scoring rubric the pipeline expects.
2. **`review-release-notes` multi-model QA pass was skipped.** That pass is designed precisely to catch under-covered clusters.
3. **Label-shaped blind spots in sub-agent slices.** #66646 is `area-infrastructure`, not `area-blazor`. Sub-agents handed one component slice drift toward "matches my area label" PRs and miss adjacent ones.
4. **PR titles don't always include the marquee API name.** #66526's title says "async form validation" — none of `AddValidationTask` / `IsValidationPending` / `IsValidationFaulted` appear. Title-keyword ranking under-rates these.
5. **No milestone cross-check.** A simple `gh api /search/issues milestone:11.0-preview` sweep against the draft would have surfaced #66526 and #66646 immediately. Not part of the current workflow.

## Proposed skill changes

1. `release-notes/SKILL.md` — add a mandatory milestone cross-check step: pull the full PR list for the milestone and reconcile against the draft's covered PR set before the QA pass.
2. `generate-features/SKILL.md` — add explicit cluster-enumeration guidance: when a cluster (e.g., "Blazor validation") is identified, enumerate every PR in `changes.json` whose title, body, or labels intersect that cluster before scoring.
3. `feature-scoring.md` — document the label-myopia anti-pattern: cross-cutting features often live under `area-infrastructure`, `area-mvc`, etc., not the obvious component area.
4. `release-notes/SKILL.md` — require sub-agents receive the full feature JSON (description, scoring rationale, body excerpt), not a slim `{id, title, url, labels}` projection.

## Validation lesson (orthogonal to the skill changes)

Building working samples for each drafted feature against the actual preview build flushes missing APIs out almost immediately: a sample that needs `AddValidationLocalization()` or `EditContext.AddValidationTask` fails to compile when those features aren't documented yet. Consider building this in as a draft validation step.

## What was already fixed in Preview 5

The three misses were patched directly into the Preview 5 draft (PR #10422, commits `8e76e667` + `9d52bc6d`). This issue tracks the workflow-level fixes so Preview 6 doesn't repeat the pattern.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.