getsentry / getsentry/XcodeBuildMCP
[Feature]: UI automation on physical devices via CoreDevice HID (pymobiledevice3)
- Lingua principale
- TypeScript
- Stelle
- 6.4k
- Fork
- 319
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
### Summary
The `device` workflow can build, install, launch, stop and test on a physical device, but has no input or capture tools — `ui-automation` is simulator-only by design ("UI automation and accessibility testing tools for iOS simulators"). That leaves agent-driven verification on real hardware out of reach, even though the device itself exposes everything needed.
I'd like to propose adding device-scoped `tap` / `swipe` / `type` / `button` and `screenshot` to the `device` workflow, and I'm happy to write the PR if there's appetite for it. Checking first because the design questions below are yours to answer, not mine.
### The capability is already on the device
`devicectl device info details` against an iPhone 17 Pro on iOS 27.0 lists these CoreDevice features as available:
```
HID Digitizer com.apple.coredevice.feature.remote.hid.digitizer
HID Keyboard com.apple.coredevice.feature.remote.hid.keyboard
HID Button com.apple.coredevice.feature.remote.hid.button
HID Scroll com.apple.coredevice.feature.remote.hid.scroll
Universal HID com.apple.coredevice.feature.remote.universalhid
View Device Screen com.apple.coredevice.feature.viewdevicescreen
```
Touch, keyboard, buttons, scroll and a screen stream. On iOS 26+ these sit behind a consent toggle at **Settings › Developer › UI Automation › Enable UI Automation**, which pairs with "Paired Mac computers can remotely view and control this iPhone" — this is the same plumbing Xcode 27's Device Hub uses for its remote-control view.
Apple ships no CLI verb for any of it. `devicectl` in the Xcode 27 beta has `appResize capture copy info install notification orientation pairings pasteboard process profile reboot rename settings simulate sysdiagnose uninstall` — no input subcommand. Device Hub is the only first-party client, and it's a GUI with no scripting interface.
### But it is reachable without one
[`pymobiledevice3`](https://github.com/doronz88/pymobiledevice3) implements the iOS 17+ RemoteXPC tunnel and exposes the service directly:
```bash
pymobiledevice3 developer core-device universal-hid-service tap -- 32768 32768
```
Coordinates are normalised UInt16 — `(0,0)` top-left, `(65535,65535)` bottom-right — with `drag`, `swipe` and `move` alongside `tap`, and batched gestures readable from stdin or a script file.
`devicectl device capture screenshot` already works against a physical device today, so pairing the two gives a complete see-and-touch loop **with no WebDriverAgent, no vendored Appium and no on-device runner to sign** — which I think is the reason this has looked bigger than it is.
### Design questions I'd want your call on before writing anything
1. **The Python dependency.** This is a Node project and pymobiledevice3 is a Python CLI. Detect at runtime and degrade gracefully when absent, or something stricter?
2. **Coordinates.** Simulator tools take points; the device service takes normalised UInt16. Convert inside the tool so the API matches `ui-automation`, or expose the normalised space and let callers deal with it? Converting needs a reliable source for the device's point size.
3. **Placement.** Extend `device`, or a separate `device-ui-automation` workflow so the Python requirement is opt-in?
4. **No accessibility tree.** There's no device equivalent of `snapshot_ui`, so device automation would be screenshot-driven only. Worth stating in the tool descriptions so agents don't reach for a tree that isn't there.
### Not verified on my side
- Whether the tunnel needs root in all transports (it does in some) and how that interacts with an MCP server's process.
- Whether the tunnel coexists with Xcode or Device Hub holding the same device.
Both would need answering in the PR rather than assumed.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Start by tracing the existing device workflow and simulator-only ui-automation entry points, then verify how `devicectl device capture screenshot` is currently exposed. Done means the Python dependency, coordinate handling, workflow placement, and screenshot-only physical-device automation scope are resolved and the supported actions are covered.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- ios, python, typescript
- Ambito
- mobile-dev, tooling
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Attiva
- Chiarezza
- Da chiarire
- Idoneità per principianti
- 35/100