callstack / callstack/agent-device
Improve iOS keyboard return reliability and perf without English label probing
- Dominant language
- TypeScript
- Stars
- 4.6k
- Forks
- 299
- Avg merge
- 10h 17m
- Merged PRs (30d)
- 515
Description
> *This was generated by AI during triage.*
## Problem
The iOS `keyboard return` path is both slower than nearby keyboard commands and biased toward English keyboard labels.
Current flow:
- `src/core/dispatch.ts` maps `keyboard enter` / `keyboard return` to the XCTest runner command `keyboardReturn`.
- `ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift` calls `tapKeyboardReturnControl(app:)` before trying `app.typeText(XCUIKeyboardKey.return.rawValue)`.
- `tapKeyboardReturnControl` probes English labels in order: `return`, `Return`, `Enter`, `Go`, `Search`, `Next`, `Done`, `Send`, `Join`.
That has two issues:
- Non-English simulator/device languages may expose localized labels, so English label probing is not reliable as the primary path.
- In the Settings search flow, the key is likely `Search`, so the runner pays several failed XCTest element lookups before the successful one. The updated iOS perf harness measured `keyboard return` at about `3833ms` wall median / `3583ms` daemon median, compared with `type` at about `2970ms` wall / `2752ms` daemon.
Perf report with the new harness row:
- `/private/tmp/agent-device-perf-pr643-series-full/perf-ios-2026-06-01T14-44-56-344Z.md`
## Likely Approach
Prefer a language-neutral return-key path before English label probing:
1. If the iOS software keyboard is visible, try `app.typeText(XCUIKeyboardKey.return.rawValue)` first inside the existing Objective-C exception catcher.
2. If that throws, try the existing single text-entry fallback: find the single text input, tap it, then `typeText(.return)`.
3. Keep visible keyboard-control tapping as a fallback, but do not make English label lookup the primary path.
4. If label fallback remains, consider querying visible keyboard buttons/keys once and matching in memory rather than issuing repeated named XCTest queries for each candidate label.
## Acceptance Criteria
- `keyboard return` still works for common iOS return-key variants such as Return, Search, Go, Next, Done, and Send.
- Add or update runner-level/manual validation notes for at least one non-English simulator language so the behavior is not English-label-dependent.
- Run the iOS perf harness and compare `keyboard return` against `type`; the goal is to remove obvious label-probing overhead, not to make keyboard return faster than text entry in every run.
- Preserve the existing unsupported-operation behavior when no keyboard/focused text input can accept return.
## Relevant Code
- `src/core/dispatch.ts` — `handleIosKeyboardCommand` sends `keyboardReturn`.
- `ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift` — runner command dispatch for `.keyboardReturn`.
- `ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift` — `pressKeyboardReturn`, `tapKeyboardReturnControl`, and `singleTextEntryElement`.
Contributor guide
Assessment
This issue has not been assessed yet.