microsoft / microsoft/winappCli

[Feature]: Publish sample indexes upstream (WinUI Gallery + Community Toolkit) so find-ui stops scraping

Open
#703 3 comments 0 reactions 1 assignee Claimed by @Jaylyn-Barbee View on GitHub
enhancement
Dominant language
C#
Stars
1.3k
Forks
80
Avg merge
3d 6h
Merged PRs (30d)
51

Description

### Is your feature request related to a problem? Please describe.

`winapp find-ui` (#681) builds its corpus by scraping three upstream repos. The cost of that scraping is wildly uneven, and it comes down entirely to **what each repo publishes**:

| Source | Publishes | Fetcher size |
|---|---|---|
| `microsoft/microsoft-ui-reactor` | [`reactor-search-index.json`](https://github.com/microsoft/microsoft-ui-reactor/blob/main/samples/ReactorGallery/reactor-search-index.json) — metadata **and sample code** | **7 KB** (`ReactorFetcher.cs`) |
| `microsoft/WinUI-Gallery` | `ControlInfoData.json` — metadata only | **43 KB** (`GalleryFetcher.cs`) |
| `CommunityToolkit/Windows` | nothing published | **54 KB** (`ToolkitFetcher.cs`) |

Reactor's fetcher is ~6x smaller for the same job, purely because the Reactor team publishes a purpose-built index. Gallery and Toolkit require us to reverse-engineer repo layout, split samples out of XAML pages, clean code-behind, fold `#if` branches, re-balance tags, and truncate — roughly **1,450 lines of parsing** that exists only to reconstruct data the upstream repos already have.

This is not just a size problem. It produces **wrong output**:

- **9 of 321 Gallery scenarios (~3%) emit structurally malformed XAML** that fails to parse. Verified against both the current corpus and the original `winui-search` snapshot in `microsoft/win-dev-skills` — identical results, so this is inherent to scraping, not a porting bug:
```
xamlresources-3, appbarbutton-5, swipecontrol-4, swipecontrol-5, listview-8,
infobadge-2, selectorbar-3, themeshadow-1, richtextblock-3
```
Example — `gallery-appbarbutton-5`, a short, **non-truncated** sample:
```xml





```
- Every upstream layout change breaks us. `CacheVersion.cs` is on version **18**, and the bump history is almost entirely upstream churn (e.g. `"17" — WinUI-Gallery moved + reformatted its samples`).
- Toolkit control names are maintained by hand in a **34-entry `SampleOverrides` table**, so a newly added Toolkit sample gets a *guessed* name until someone notices.

Since `find-ui` is aimed at AI agents, a snippet that looks authoritative but doesn't compile or parse is worse than no result at all.

This also isn't a winapp-only problem: every WinUI agent/tool that wants sample data re-scrapes these same repos and independently re-derives these same bugs.

### Describe the solution you'd like

Submit PRs to the two upstream repos so each publishes a machine-readable sample index, the way `microsoft-ui-reactor` already does.

**1. `CommunityToolkit/Windows` — lowest effort, biggest win.**

The Toolkit **already generates exactly the index we need**. `ToolkitSampleMetadataGenerator` (in `CommunityToolkit/Tooling-Windows-Submodule`) stitches together:
- YAML frontmatter on every `components/*/samples/*.md` (`title`, `description`, `keywords`, `category`, `subcategory`)
- `> [!SAMPLE XSample]` markers linking docs to sample classes
- `[ToolkitSample(id:, title:, description:)]` attributes on the sample classes

…into `ToolkitSampleMetadata` at build time, with source-generator diagnostics when the frontmatter is malformed. It just never leaves the compiler — it only materializes inside the gallery app's binary.

**Ask:** also emit that metadata as a JSON artifact (repo file, release asset, or docs-site output). No new data modeling and no new validation — the assembly step already exists and is already enforced.

**2. `microsoft/WinUI-Gallery` — additive schema change.**

`WinUIGallery/SampleSupport/Data/ControlInfoData.json` already carries `UniqueId`, `Title`, `Subtitle`, `Description`, `ApiNamespace`, `RelatedControls`, and `Docs[]`. What's missing is the sample code, which lives in separate `Samples/{UniqueId}/{SampleDefinition}` `.txt` bundles.

**Ask:** inline those bundle contents (`--- header` / `--- xaml` / `--- c#`) into `ControlInfoData.json`, or emit a sibling index that does. The repo already maintains `ControlInfoDataSchema.json`, so there's a natural place to version the addition.

**Outcome.** Either change makes that source Reactor-shaped: its fetcher collapses from ~40-50 KB to ~100 lines, and the malformed-output class of bug disappears **by construction**, because nothing is being guessed.

### Additional context

- Blocked on nobody — this is follow-up work after #681 merges. #681 should ship the local mitigations (validate snippets at the corpus boundary; parse `[ToolkitSample(...)]` instead of stripping it) independently of upstream timelines.
- Suggested order: open the Toolkit issue/PR first — their data is already assembled and validated, so it's the shortest path to a proof point that helps the Gallery conversation.
- Once an upstream index exists, `GalleryFetcher.cs` / `ToolkitFetcher.cs` and most of `CacheVersion.cs`'s bump history can be deleted outright.
- Prior art within this codebase: `ReactorFetcher.cs` + `DATA_SOURCES.md` in `microsoft/win-dev-skills`, which states it directly — *"The Reactor team owns a purpose-built search index we consume directly, so there is no scraping or per-file cleaning."*

## Phases

- #808 — Phase 0: define the shared sample index contract
- #809 — Phase 1: WinUI-Gallery publishes a sample index (and we delete `GalleryFetcher`)
- #810 — Phase 2: CommunityToolkit publishes a sample index (and we delete `ToolkitFetcher`)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.