getsentry / getsentry/XcodeBuildMCP

[Feature]: UI automation on physical devices via CoreDevice HID (pymobiledevice3)

未关闭
#519 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
TypeScript
星标
6.4k
派生
319
PR 合并指标
30 天内没有已合并 PR

描述

### Summary

The `device` workflow can build, install, launch, stop and test on a physical device, but has no input or capture tools — `ui-automation` is simulator-only by design ("UI automation and accessibility testing tools for iOS simulators"). That leaves agent-driven verification on real hardware out of reach, even though the device itself exposes everything needed.

I'd like to propose adding device-scoped `tap` / `swipe` / `type` / `button` and `screenshot` to the `device` workflow, and I'm happy to write the PR if there's appetite for it. Checking first because the design questions below are yours to answer, not mine.

### The capability is already on the device

`devicectl device info details` against an iPhone 17 Pro on iOS 27.0 lists these CoreDevice features as available:

```
HID Digitizer com.apple.coredevice.feature.remote.hid.digitizer
HID Keyboard com.apple.coredevice.feature.remote.hid.keyboard
HID Button com.apple.coredevice.feature.remote.hid.button
HID Scroll com.apple.coredevice.feature.remote.hid.scroll
Universal HID com.apple.coredevice.feature.remote.universalhid
View Device Screen com.apple.coredevice.feature.viewdevicescreen
```

Touch, keyboard, buttons, scroll and a screen stream. On iOS 26+ these sit behind a consent toggle at **Settings › Developer › UI Automation › Enable UI Automation**, which pairs with "Paired Mac computers can remotely view and control this iPhone" — this is the same plumbing Xcode 27's Device Hub uses for its remote-control view.

Apple ships no CLI verb for any of it. `devicectl` in the Xcode 27 beta has `appResize capture copy info install notification orientation pairings pasteboard process profile reboot rename settings simulate sysdiagnose uninstall` — no input subcommand. Device Hub is the only first-party client, and it's a GUI with no scripting interface.

### But it is reachable without one

[`pymobiledevice3`](https://github.com/doronz88/pymobiledevice3) implements the iOS 17+ RemoteXPC tunnel and exposes the service directly:

```bash
pymobiledevice3 developer core-device universal-hid-service tap -- 32768 32768
```

Coordinates are normalised UInt16 — `(0,0)` top-left, `(65535,65535)` bottom-right — with `drag`, `swipe` and `move` alongside `tap`, and batched gestures readable from stdin or a script file.

`devicectl device capture screenshot` already works against a physical device today, so pairing the two gives a complete see-and-touch loop **with no WebDriverAgent, no vendored Appium and no on-device runner to sign** — which I think is the reason this has looked bigger than it is.

### Design questions I'd want your call on before writing anything

1. **The Python dependency.** This is a Node project and pymobiledevice3 is a Python CLI. Detect at runtime and degrade gracefully when absent, or something stricter?
2. **Coordinates.** Simulator tools take points; the device service takes normalised UInt16. Convert inside the tool so the API matches `ui-automation`, or expose the normalised space and let callers deal with it? Converting needs a reliable source for the device's point size.
3. **Placement.** Extend `device`, or a separate `device-ui-automation` workflow so the Python requirement is opt-in?
4. **No accessibility tree.** There's no device equivalent of `snapshot_ui`, so device automation would be screenshot-driven only. Worth stating in the tool descriptions so agents don't reach for a tree that isn't there.

### Not verified on my side

- Whether the tunnel needs root in all transports (it does in some) and how that interacts with an MCP server's process.
- Whether the tunnel coexists with Xcode or Device Hub holding the same device.

Both would need answering in the PR rather than assumed.

贡献指南

这个仓库没有索引到贡献指南

调研方向

Start by tracing the existing device workflow and simulator-only ui-automation entry points, then verify how `devicectl device capture screenshot` is currently exposed. Done means the Python dependency, coordinate handling, workflow placement, and screenshot-only physical-device automation scope are resolved and the supported actions are covered.

由索引模型根据 Issue 内容生成。

评估

技术栈
ios, python, typescript
领域
mobile-dev, tooling
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
活跃
描述清晰度
需要澄清
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。