getsentry / getsentry/XcodeBuildMCP

[Feature]: UI automation on physical devices via CoreDevice HID (pymobiledevice3)

Abierto
#519 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
TypeScript
Estrellas
6.4k
Forks
319
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

### Summary

The `device` workflow can build, install, launch, stop and test on a physical device, but has no input or capture tools — `ui-automation` is simulator-only by design ("UI automation and accessibility testing tools for iOS simulators"). That leaves agent-driven verification on real hardware out of reach, even though the device itself exposes everything needed.

I'd like to propose adding device-scoped `tap` / `swipe` / `type` / `button` and `screenshot` to the `device` workflow, and I'm happy to write the PR if there's appetite for it. Checking first because the design questions below are yours to answer, not mine.

### The capability is already on the device

`devicectl device info details` against an iPhone 17 Pro on iOS 27.0 lists these CoreDevice features as available:

```
HID Digitizer com.apple.coredevice.feature.remote.hid.digitizer
HID Keyboard com.apple.coredevice.feature.remote.hid.keyboard
HID Button com.apple.coredevice.feature.remote.hid.button
HID Scroll com.apple.coredevice.feature.remote.hid.scroll
Universal HID com.apple.coredevice.feature.remote.universalhid
View Device Screen com.apple.coredevice.feature.viewdevicescreen
```

Touch, keyboard, buttons, scroll and a screen stream. On iOS 26+ these sit behind a consent toggle at **Settings › Developer › UI Automation › Enable UI Automation**, which pairs with "Paired Mac computers can remotely view and control this iPhone" — this is the same plumbing Xcode 27's Device Hub uses for its remote-control view.

Apple ships no CLI verb for any of it. `devicectl` in the Xcode 27 beta has `appResize capture copy info install notification orientation pairings pasteboard process profile reboot rename settings simulate sysdiagnose uninstall` — no input subcommand. Device Hub is the only first-party client, and it's a GUI with no scripting interface.

### But it is reachable without one

[`pymobiledevice3`](https://github.com/doronz88/pymobiledevice3) implements the iOS 17+ RemoteXPC tunnel and exposes the service directly:

```bash
pymobiledevice3 developer core-device universal-hid-service tap -- 32768 32768
```

Coordinates are normalised UInt16 — `(0,0)` top-left, `(65535,65535)` bottom-right — with `drag`, `swipe` and `move` alongside `tap`, and batched gestures readable from stdin or a script file.

`devicectl device capture screenshot` already works against a physical device today, so pairing the two gives a complete see-and-touch loop **with no WebDriverAgent, no vendored Appium and no on-device runner to sign** — which I think is the reason this has looked bigger than it is.

### Design questions I'd want your call on before writing anything

1. **The Python dependency.** This is a Node project and pymobiledevice3 is a Python CLI. Detect at runtime and degrade gracefully when absent, or something stricter?
2. **Coordinates.** Simulator tools take points; the device service takes normalised UInt16. Convert inside the tool so the API matches `ui-automation`, or expose the normalised space and let callers deal with it? Converting needs a reliable source for the device's point size.
3. **Placement.** Extend `device`, or a separate `device-ui-automation` workflow so the Python requirement is opt-in?
4. **No accessibility tree.** There's no device equivalent of `snapshot_ui`, so device automation would be screenshot-driven only. Worth stating in the tool descriptions so agents don't reach for a tree that isn't there.

### Not verified on my side

- Whether the tunnel needs root in all transports (it does in some) and how that interacts with an MCP server's process.
- Whether the tunnel coexists with Xcode or Device Hub holding the same device.

Both would need answering in the PR rather than assumed.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Start by tracing the existing device workflow and simulator-only ui-automation entry points, then verify how `devicectl device capture screenshot` is currently exposed. Done means the Python dependency, coordinate handling, workflow placement, and screenshot-only physical-device automation scope are resolved and the supported actions are covered.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
ios, python, typescript
Área
mobile-dev, tooling
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Activo
Claridad
Necesita aclaración
Aptitud para principiantes
35/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.