getsentry / getsentry/XcodeBuildMCP

[Feature]: UI automation on physical devices via CoreDevice HID (pymobiledevice3)

Aperta
#519 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
TypeScript
Stelle
6.4k
Fork
319
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

### Summary

The `device` workflow can build, install, launch, stop and test on a physical device, but has no input or capture tools — `ui-automation` is simulator-only by design ("UI automation and accessibility testing tools for iOS simulators"). That leaves agent-driven verification on real hardware out of reach, even though the device itself exposes everything needed.

I'd like to propose adding device-scoped `tap` / `swipe` / `type` / `button` and `screenshot` to the `device` workflow, and I'm happy to write the PR if there's appetite for it. Checking first because the design questions below are yours to answer, not mine.

### The capability is already on the device

`devicectl device info details` against an iPhone 17 Pro on iOS 27.0 lists these CoreDevice features as available:

```
HID Digitizer com.apple.coredevice.feature.remote.hid.digitizer
HID Keyboard com.apple.coredevice.feature.remote.hid.keyboard
HID Button com.apple.coredevice.feature.remote.hid.button
HID Scroll com.apple.coredevice.feature.remote.hid.scroll
Universal HID com.apple.coredevice.feature.remote.universalhid
View Device Screen com.apple.coredevice.feature.viewdevicescreen
```

Touch, keyboard, buttons, scroll and a screen stream. On iOS 26+ these sit behind a consent toggle at **Settings › Developer › UI Automation › Enable UI Automation**, which pairs with "Paired Mac computers can remotely view and control this iPhone" — this is the same plumbing Xcode 27's Device Hub uses for its remote-control view.

Apple ships no CLI verb for any of it. `devicectl` in the Xcode 27 beta has `appResize capture copy info install notification orientation pairings pasteboard process profile reboot rename settings simulate sysdiagnose uninstall` — no input subcommand. Device Hub is the only first-party client, and it's a GUI with no scripting interface.

### But it is reachable without one

[`pymobiledevice3`](https://github.com/doronz88/pymobiledevice3) implements the iOS 17+ RemoteXPC tunnel and exposes the service directly:

```bash
pymobiledevice3 developer core-device universal-hid-service tap -- 32768 32768
```

Coordinates are normalised UInt16 — `(0,0)` top-left, `(65535,65535)` bottom-right — with `drag`, `swipe` and `move` alongside `tap`, and batched gestures readable from stdin or a script file.

`devicectl device capture screenshot` already works against a physical device today, so pairing the two gives a complete see-and-touch loop **with no WebDriverAgent, no vendored Appium and no on-device runner to sign** — which I think is the reason this has looked bigger than it is.

### Design questions I'd want your call on before writing anything

1. **The Python dependency.** This is a Node project and pymobiledevice3 is a Python CLI. Detect at runtime and degrade gracefully when absent, or something stricter?
2. **Coordinates.** Simulator tools take points; the device service takes normalised UInt16. Convert inside the tool so the API matches `ui-automation`, or expose the normalised space and let callers deal with it? Converting needs a reliable source for the device's point size.
3. **Placement.** Extend `device`, or a separate `device-ui-automation` workflow so the Python requirement is opt-in?
4. **No accessibility tree.** There's no device equivalent of `snapshot_ui`, so device automation would be screenshot-driven only. Worth stating in the tool descriptions so agents don't reach for a tree that isn't there.

### Not verified on my side

- Whether the tunnel needs root in all transports (it does in some) and how that interacts with an MCP server's process.
- Whether the tunnel coexists with Xcode or Device Hub holding the same device.

Both would need answering in the PR rather than assumed.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Start by tracing the existing device workflow and simulator-only ui-automation entry points, then verify how `devicectl device capture screenshot` is currently exposed. Done means the Python dependency, coordinate handling, workflow placement, and screenshot-only physical-device automation scope are resolved and the supported actions are covered.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
ios, python, typescript
Ambito
mobile-dev, tooling
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Attiva
Chiarezza
Da chiarire
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.