getsentry / getsentry/XcodeBuildMCP

[Feature]: UI automation on physical devices via CoreDevice HID (pymobiledevice3)

Đang mở
#519 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
TypeScript
Star
6.4k
Fork
319
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

### Summary

The `device` workflow can build, install, launch, stop and test on a physical device, but has no input or capture tools — `ui-automation` is simulator-only by design ("UI automation and accessibility testing tools for iOS simulators"). That leaves agent-driven verification on real hardware out of reach, even though the device itself exposes everything needed.

I'd like to propose adding device-scoped `tap` / `swipe` / `type` / `button` and `screenshot` to the `device` workflow, and I'm happy to write the PR if there's appetite for it. Checking first because the design questions below are yours to answer, not mine.

### The capability is already on the device

`devicectl device info details` against an iPhone 17 Pro on iOS 27.0 lists these CoreDevice features as available:

```
HID Digitizer com.apple.coredevice.feature.remote.hid.digitizer
HID Keyboard com.apple.coredevice.feature.remote.hid.keyboard
HID Button com.apple.coredevice.feature.remote.hid.button
HID Scroll com.apple.coredevice.feature.remote.hid.scroll
Universal HID com.apple.coredevice.feature.remote.universalhid
View Device Screen com.apple.coredevice.feature.viewdevicescreen
```

Touch, keyboard, buttons, scroll and a screen stream. On iOS 26+ these sit behind a consent toggle at **Settings › Developer › UI Automation › Enable UI Automation**, which pairs with "Paired Mac computers can remotely view and control this iPhone" — this is the same plumbing Xcode 27's Device Hub uses for its remote-control view.

Apple ships no CLI verb for any of it. `devicectl` in the Xcode 27 beta has `appResize capture copy info install notification orientation pairings pasteboard process profile reboot rename settings simulate sysdiagnose uninstall` — no input subcommand. Device Hub is the only first-party client, and it's a GUI with no scripting interface.

### But it is reachable without one

[`pymobiledevice3`](https://github.com/doronz88/pymobiledevice3) implements the iOS 17+ RemoteXPC tunnel and exposes the service directly:

```bash
pymobiledevice3 developer core-device universal-hid-service tap -- 32768 32768
```

Coordinates are normalised UInt16 — `(0,0)` top-left, `(65535,65535)` bottom-right — with `drag`, `swipe` and `move` alongside `tap`, and batched gestures readable from stdin or a script file.

`devicectl device capture screenshot` already works against a physical device today, so pairing the two gives a complete see-and-touch loop **with no WebDriverAgent, no vendored Appium and no on-device runner to sign** — which I think is the reason this has looked bigger than it is.

### Design questions I'd want your call on before writing anything

1. **The Python dependency.** This is a Node project and pymobiledevice3 is a Python CLI. Detect at runtime and degrade gracefully when absent, or something stricter?
2. **Coordinates.** Simulator tools take points; the device service takes normalised UInt16. Convert inside the tool so the API matches `ui-automation`, or expose the normalised space and let callers deal with it? Converting needs a reliable source for the device's point size.
3. **Placement.** Extend `device`, or a separate `device-ui-automation` workflow so the Python requirement is opt-in?
4. **No accessibility tree.** There's no device equivalent of `snapshot_ui`, so device automation would be screenshot-driven only. Worth stating in the tool descriptions so agents don't reach for a tree that isn't there.

### Not verified on my side

- Whether the tunnel needs root in all transports (it does in some) and how that interacts with an MCP server's process.
- Whether the tunnel coexists with Xcode or Device Hub holding the same device.

Both would need answering in the PR rather than assumed.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Hướng nghiên cứu

Start by tracing the existing device workflow and simulator-only ui-automation entry points, then verify how `devicectl device capture screenshot` is currently exposed. Done means the Python dependency, coordinate handling, workflow placement, and screenshot-only physical-device automation scope are resolved and the supported actions are covered.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
ios, python, typescript
Lĩnh vực
mobile-dev, tooling
Loại issue
Tính năng
Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Cần làm rõ
Mức phù hợp với người mới
35/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.