microsoft / microsoft/winappCli

[Feature]: `winapp ui` needs scoped/typed queries, text attributes and a batch mode to back a real UI suite

Open
#798 4 comments 0 reactions 1 assignee Claimed by @nmetulev View on GitHub
enhancement
Dominant language
C#
Stars
1.3k
Forks
80
Avg merge
3d 6h
Merged PRs (30d)
51

Description

### What problem are you trying to solve?

The [AI-assisted testing docs](https://learn.microsoft.com/en-us/windows/apps/develop/ai-assisted/testing) recommend generating xUnit UI tests that "use `winapp ui` commands as the interaction layer, so they run without a separate UI automation framework." I tried to adopt exactly that for an existing, non-trivial suite and could not.

**The app and the suite.** A WinUI 3 mail/calendar client. 73 cases driven from PowerShell against in-process UI Automation (`System.Windows.Automation`). Environment: Windows 11 ARM64 (10.0.26200), `winapp` **0.6.1**, app is unpackaged Debug WinUI 3. Every number below is measured on that machine against that app.

Four capability gaps and one silent-failure bug. Three of the four produce a **green assertion for something that is not true**, which is worse than a missing feature.

**1 — No scoped queries.** There is no way to search *inside* an element; every selector resolves against the whole window. A mail row is a `ListViewItem` whose `Name` is the composed announcement ("MyApp, Welcome to MyApp, 09:06, Unread"), containing separate `Text` elements for sender/subject/time. Asking for the subject text:

```
winapp ui get-property "Welcome to MyApp" -a --json
```

returns the **ListItem** (`"ControlType": "ListItem"`, `"ClassName": "ListViewItem"`) with `"BoundingRectangle": "506,184,905,94"`. The `Text` element actually named that has bounds `708,196,573,38`. A geometry assertion written this way measures the row instead of the text **and passes**.

**2 — No control-type filter.** The same repro's other half: matching on name alone also matches the inert `TextBlock` *inside* a control, which carries the same `Name` and supports no patterns. In our harness "always pass a control type for anything you intend to press" is a hard rule, because a bare name match once returned a reading-pane label and reported a green PASS for a context-menu item that did not exist. `invoke` mitigates this by trying several patterns in order, but that does not let a test *assert* it found a `Button` rather than a label.

**3 — No TextPattern text attributes, in particular `FontWeight`.**

```
winapp ui get-property "" -a -p FontWeight --json
→ { "properties": { "FontWeight": null } }
```

For the same element, in-process:

```csharp
((TextPattern)el.GetCurrentPattern(TextPattern.Pattern))
.DocumentRange.GetAttributeValue(TextPattern.FontWeightAttribute) // → 600
```

A WinUI `TextBlock` exposes `TextPattern`, so the weight the user is actually looking at is readable. This is the single most valuable assertion in our suite: *"is this unread row actually bold?"* It caught a shipped bug where the XAML binding was correct and the bound property was simply never assigned — invisible to unit tests, invisible to a screenshot, and invisible to every property `winapp ui` exposes today.

**4 — Every verb is a fresh process.** Median of 6–8 runs:

| | median |
|---|---|
| `winapp --version` (process start alone, no app attach) | **161 ms** |
| `winapp ui get-property -a -p Name --json` (attached) | **456 ms** |
| in-process equivalent (find element + read property) | **5–7 ms** |

~70–90× per assertion. Our 73 cases make roughly 230 primitive calls *statically*, many inside `foreach` loops over rows, so the real count runs into the hundreds. At 456 ms each, a faithful port would add minutes to a suite we were actively trying to make faster. This is the gap that decides the docs' own recommendation: the xUnit integration is viable for toy tests but not for a suite that gates a release.

~**5 — `inspect --json` nests one level deeper than consumers expect.**~ **✅ Fixed in https://github.com/microsoft/win-dev-skills/pull/175** The tree sits under `windows[].elements[]`. Code reading `.elements` at the **root** silently gets `null` — and in an audit loop that is a PASS over nothing examined. The `winui-ui-testing` skill's accessibility-audit snippet has exactly this shape:

```powershell
$allElements = (winapp ui inspect -a $AppPid --interactive --json | ConvertFrom-Json).elements
```

### Proposed solution

Four CLI additions and one schema fix. Items 1–3 are what a suite needs in order to assert without false passes; item 4 is what it needs to finish in reasonable time.

1. **`--root `** (or `--within`) on `search` / `get-property` / `get-value` / `wait-for`, scoping the query to a subtree. Highest-value item of the five — it is what makes an assertion about a templated list item expressible at all.

2. **`--type Button|Text|ListItem|MenuItem|…`** on `search` and `wait-for`, so a test can require the control type it means rather than accepting whatever happens to share the name.

3. **TextPattern attributes via `get-property`** — `FontWeight`, `FontName`, `FontSize`, `ForegroundColor`, `IsItalic`, `StrikethroughStyle` — or a dedicated `get-text-attribute` verb if they do not belong beside UIA element properties.

4. **A persistent mode**: `winapp ui repl` reading verbs from stdin and writing one JSON result per line, or `winapp ui --batch commands.txt`. One attach, many verbs. This is what would make the documented xUnit integration practical rather than illustrative.

5. ~**Document the `inspect --json` schema**, or add a top-level `elements` alias.~ **✅ fixed in https://github.com/microsoft/win-dev-skills/pull/175** A silent `null` is the wrong failure mode for a shape people will copy, and fixing the one snippet in the skill does not cover it — any consumer of `inspect --json` hits the same thing (see Additional context re microsoft/win-dev-skills#139).

### Alternatives considered

**Porting the suite anyway and accepting the cost.** Rejected on item 4: several hundred invocations at 456 ms is minutes of pure process startup, added to a suite whose runtime we were trying to reduce.

**Working around the scoping gap by dumping the tree and filtering client-side.** `inspect --json` with `--depth` can return a subtree, so a script could in principle dump once and select in PowerShell. That is re-implementing the query engine on the client, it still pays one process per dump, and it does nothing for items 2 or 3.

**Keeping our own harness — what we actually did.** We stayed on in-process UIA and attacked the app-launch cost instead (one launch shared across suites needing the same fixture), which took the suite from 578 s to 275 s with no loss of coverage. That closed the immediate problem, but leaves us maintaining a harness we would rather not own.

**Continuing to use `winapp ui` for what it is already better at.** We do, and will. It opens WinUI context flyouts first try where our own synthetic right-click could not, because it is DPI-aware and clicks the real physical point. This request is about the *assertion* layer specifically, not about replacing the tool.

### Additional context

**Upstream docs this came from:** — specifically "Write automated tests" and "Run tests in CI", which is the workflow being attempted here.

**Related issues.** I checked all 41 issues and 99 comments in this repo; none of items 1–4 appears anywhere, so this is not a duplicate. Three are adjacent:

- **#139** — the AxeWindowsCLI accessibility recipe. It would replace the guidance containing the snippet quoted in item 5, so that specific example may disappear on its own. The underlying schema surprise would not: any consumer of `inspect --json` hits it, and it fails silently rather than erroring.
- **#68** — session feedback on these same skills, including scaling `winui-ui-testing` output to feature surface. Complementary rather than overlapping: that one is about *how many* tests get generated, this one about what an assertion can express and what it costs.
- **#122** — keeping skills current as the CLI grows. Items 1–4 are CLI-side capability requests, so they would land there first and reach the skills through that process.

**Offer.** Happy to re-run any of these measurements, share the harness, or test a prerelease build with `--root` in it.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.