microsoft / microsoft/winappCli
[Feature]: `winapp find-ui` — semantic search for UI controls, samples & docs
- Dominant language
- C#
- Stars
- 1.3k
- Forks
- 80
- Avg merge
- 3d 6h
- Merged PRs (30d)
- 51
Description
> **Depends on #651** — the lexical/fuzzy port of `winui-search` into `winapp` (full v1). This issue adds the **semantic layer** on top of that ported foundation.
## Is your feature request related to a problem? Please describe.
When building a Windows UI, a developer (or agent) has to already know which control to use and where to find a working example. Discovery today means leaving the CLI and manually hunting through the WinUI Gallery, the Windows Community Toolkit gallery, and docs. For an agent this is especially costly — it burns turns guessing at control names and APIs instead of getting a grounded, working snippet.
`winapp` should be able to answer "what control does X, and show me a real example" from the command line, in natural language.
## Current state
- Control/sample discovery lives in the separate **`winui-search`** tool (in the **microsoft/win-dev-skills** repo), which does **fuzzy/lexical** matching over a **set of controls/gallery samples** — you have to roughly know the control name; natural-language intent ("a list with swipeable rows") doesn't resolve well.
- It covers controls + gallery samples today, but **lexically and without docs** — this issue makes it semantic and folds docs into the same corpus.
- Note: this is distinct from `winapp ui search`, which searches a *running app's* UI tree (UI automation) — unrelated to control/sample discovery.
## Describe the solution you'd like
Add a **`winapp find-ui ""`** command: natural-language, **semantic** search across WinUI **controls, samples, and docs** that returns a working code example plus where it came from. This is **WinUI-scoped** (its corpus is WinUI + Windows Community Toolkit controls/samples/docs), so it's nested/scoped as a WinUI capability.
- **Natural-language query** — "a card with an image and title", "tabbed layout", not just exact control names.
- **Semantic matching** — ranks by intent; the existing fuzzy/lexical search stays as the fast exact-name path.
- **Controls + samples + docs** — searches the WinUI Gallery, Windows Community Toolkit gallery, and docs, with filter options to narrow (e.g. `--source toolkit`).
- **Agent-friendly output** — `--json` with a structured result (control, snippet, source, link) so an agent gets a grounded example in one shot.
- **Extensible** — leave a seam for additional authoring styles later (e.g. Reactor controls).
Replaces the `winui-search` tool (from **microsoft/win-dev-skills**), upgrading it from fuzzy to semantic.
## Proposed architecture
Semantic search over **two kinds of corpus**, unified in **one shared vector space** (shared with `find-api`) so a single query searches them together:
- **Global corpus — controls, samples, and docs** (same for every user): gathered and embedded **centrally at release**, delivered as a **download-only, fetch-and-cached vector index**. Refreshes by re-publishing server-side; the CLI pulls updates when online. No user data is ever uploaded.
- **Query embedding — on-device** with one small bundled model. The query is compared locally against the cached vectors; the existing fuzzy/lexical search stays as the fast exact-name path.
```
winapp find-ui ""
├─ ensure global control/sample/doc index (bundled baseline + fetch/refresh when online)
├─ embed the query locally with the bundled model
├─ cosine match against the cached vectors (lexical/fuzzy = fast exact-name path)
└─ return ranked results (control, snippet, source, link) [+ --json]
```
- **Payload minimized:** the only heavy bundled artifact is the shared embedding model. The global index is fetched-and-cached, not shipped whole; only a small baseline ships for offline.
- **Shared model / one vector space:** `find-ui` and `find-api` use one embedding model, so UI + API + docs + sample hits can eventually merge into one ranked answer (fewer agent turns, fewer tokens).
- **Download-only cloud:** the service serves *public* prebuilt vectors to the client — it never receives the query or user data; embedding + comparison happen locally.
- **Feasibility:** a local PoC with a 384-dim embedding model ranked the correct control from natural-language intent (e.g. "swipe rows to delete" → SwipeControl), with a sub-2 MB index and ~20 ms warm queries.
- **The main thing to approve:** `find-ui` = bundled shared model + a fetch-and-cached global index over controls, samples, and docs.
## Open questions
- [ ] Payload budget for the bundled model (~30–40 MB quantized)
- [ ] Global index delivery - how much ships "baseline" and what do we leave to fetch.
- [ ] How docs/samples are gathered and embedded centrally (sources, chunking, release pipeline).
- [ ] Command placement for a WinUI-scoped capability. Existing commands are either flat top-level (`init`, `run`, `sign`) or grouped by **capability noun** (`cert …`, `manifest …`, `ui …`) — never by framework; framework differences are handled by flags (e.g. `init --setup-sdks none`). So: flat `winapp find-ui` (WinUI-scoped by its corpus) vs. a grouping, given there's no framework-namespace precedent?
## Additional context
Part of the single-`winapp`-CLI direction (new top-level command; WinUI-aware). Migrates the existing **`winui-search`** tool from **microsoft/win-dev-skills** into `winapp`, upgrading it from fuzzy to semantic. Sibling of `winapp find-api` (semantic API search) — the two share an embedding model/vector space toward a future unified semantic index across UI, docs, and APIs. Future function `winapp ask ` once each has been established?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.