Add a golden-set evaluation for Aspire docs search relevance
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
## Summary
Aspire CLI has a docs search command that ranks pages from aspire.dev. We should add a small, repeatable relevance evaluation so future ranking changes can be judged with data instead of only spot checks.
This issue is about **measuring search quality**, not changing the search algorithm immediately.
## Why this matters
Search can look good for one query and still fail nearby queries. For example, a ranking change might fix `whats new 13.4` but accidentally make `service discovery`, `aspire new`, or `azure container apps environment` worse.
A search relevance eval gives us a stable way to answer:
- Did the best result get better or worse?
- Did a good result appear somewhere in the first few results?
- Are we returning broad release notes or changelog pages when a dedicated doc exists?
- Are ambiguous queries being handled reasonably?
- Which query categories still fail?
## What is a "golden set"?
A golden set is a small file of realistic search queries with human-reviewed expected results.
Instead of saying every query has exactly one correct page, the file should allow multiple results with different quality ratings. That matters because docs search often has ambiguous queries where more than one page can be useful.
Example:
```json
{
"query": "whats new 13.4",
"judgments": [
{
"slug": "whats-new-in-aspire-134",
"rating": 3,
"notes": "Exact page the user asked for."
},
{
"slug": "whats-new-in-aspire-133",
"rating": 1,
"notes": "Related release notes page, but the wrong version."
},
{
"slug": "configure-azure-container-apps-environments",
"rating": 0,
"notes": "Mentions related terms but does not answer the query."
}
]
}
```
Suggested rating scale:
| Rating | Meaning | Example |
|---:|---|---|
| 3 | Perfect | The query `whats new 13.4` returns `What's new in Aspire 13.4`. |
| 2 | Useful | The query `azure container apps` returns a focused Azure Container Apps setup or configuration page. |
| 1 | Related | The query `service discovery` returns a page that mentions service discovery but is not mainly about it. |
| 0 | Irrelevant or misleading | The query `go` returns `Changelog` only because `go` appears inside `changelog`. |
## What should the eval report?
The eval should run each query through `aspire docs search` or `DocsIndexService.SearchAsync`, compare the ranked results to the golden set, and print metrics.
Definitions:
| Metric | Plain-English meaning | Why it matters |
|---|---|---|
| Top 1 accuracy | The first result is good enough. | Most users look at the first result first. |
| Recall@3 | At least one good result appears in the first 3 results. | Useful when the top result is not perfect but the answer is still visible. |
| Recall@5 | At least one good result appears in the first 5 results. | Matches the short list users commonly scan in CLI output. |
| MRR | Mean Reciprocal Rank. A good result at rank 1 scores 1.0, rank 2 scores 0.5, rank 3 scores 0.333, and missing scores 0. | Rewards putting the first good result higher. |
| NDCG@5 or NDCG@10 | Gives more credit to highly rated results and more credit when they appear near the top. | Handles graded relevance, where a perfect result should beat a merely related result. |
| Zero-result rate | How often search returns nothing. | Docs search should rarely return nothing for realistic Aspire queries. |
| Bad-top-result rate | How often the first result is rated 0. | Captures the most painful failure: search confidently returns something irrelevant. |
## Query categories to include
Start with about 50-100 queries. Include both easy and hard cases.
| Category | Why include it | Example queries |
|---|---|---|
| Release/version queries | These catch version normalization and release-note ranking bugs. | `whats new 13.4`, `what new 13.4`, `13-4 whats new`, `changelog` |
| Dedicated docs | These should strongly prefer the canonical topic page. | `service discovery`, `aspire new`, `typescript apphost`, `service defaults` |
| Azure/deployment | These pages share many repeated terms and can be noisy. | `azure container apps environment`, `deploy to azure`, `aca`, `azure app service` |
| Integrations | Many integration pages have similar structure and repeated headings. | `redis`, `postgresql`, `kafka`, `go feature flag`, `azure openai` |
| Section-level queries | These test whether search can find the right page from a common section or concept. | `connection properties`, `connection string`, `environment variables`, `health checks` |
| Acronyms and synonyms | Users often type shorter names than docs titles use. | `ACA`, `azd`, `AppHost`, `OTel`, `container apps` |
| Ambiguous/generic queries | These should allow multiple acceptable results. | `configuration`, `deployment`, `authentication`, `dashboard` |
## What the runner should output
The runner should be useful when changing search ranking. It should print:
- Overall metrics.
- Metrics by query category.
- The worst failures.
- Queries that improved or regressed compared with a baseline, if a baseline is provided.
- For each failure: query, top returned results, expected ratings, and notes.
Example failure output shape:
```text
Query: connection properties
Category: section-level
Top results:
1. Connect to MySQL with EF Core (rating: 1)
2. Connect to Redis (rating: 2)
Expected perfect result(s): any integration-specific page matching the user's intended resource
Notes: Query is ambiguous. Multiple integration docs contain the same heading.
```
## Why generated corpus tests are not enough
A generated corpus test can create queries from every page title, slug, summary, and heading, then expect the source page to come back. That is useful for broad regression detection, but it is not a real relevance benchmark.
Examples of where generated tests are misleading:
- `connection properties` appears across many integration docs, so there may be many acceptable answers.
- `connection string` appears in many setup and connection docs.
- `to azure` is too broad to identify one page.
- `connect us` can be generated from prose but is not a realistic user query.
A human-reviewed golden set can mark multiple documents as acceptable and distinguish perfect results from merely related results.
## Recent data point
As part of validating a docs search ranking change, I ran an offline generated-corpus evaluation against aspire.dev `llms-full.txt`:
- Corpus size: 5,472,074 bytes
- Parsed docs: 475
- Generated scenarios: 2,689
That generated eval showed a large improvement for the current ranking change:
| Metric | Before | After | Delta |
|---|---:|---:|---:|
| Top 1 | 31.5% | 86.5% | +55.0 pp |
| Top 3 | 43.0% | 92.2% | +49.3 pp |
| Top 5 | 48.0% | 93.5% | +45.5 pp |
| MRR | 0.387 | 0.896 | +0.509 |
This is encouraging, but it still has the limitations described above. It should not replace a human-reviewed golden set.
## Proposed acceptance criteria
- A documented golden-set format exists.
- The golden set includes representative queries from the categories above.
- The runner executes the current docs search implementation against the golden set.
- The runner reports Top 1, Recall@3, Recall@5, MRR, NDCG@5 or NDCG@10, zero-result rate, and bad-top-result rate.
- The runner prints the worst failures with enough detail to diagnose ranking problems.
- The runner can compare a branch against a baseline so ranking changes show improvements and regressions.
## Non-goals for the first version
- This does not need to block CI immediately.
- This does not need to solve every ambiguous query.
- This does not need to introduce embeddings or a new search backend.
- This does not need a huge dataset. A carefully reviewed 50-100 query set is enough to start.
Contributor guide
Assessment
This issue has not been assessed yet.