callstack / callstack/agent-device
Improve iOS keyboard return reliability and perf without English label probing
Dieses Issue hat noch niemand übernommen.
- Vorherrschende Sprache
- TypeScript
- Sterne
- 4.7k
- Forks
- 303
- Ø Merge
- 10 Std. 42 Min.
- Gemergte PRs (30 T.)
- 493
Beschreibung
> *This was generated by AI during triage.*
## Problem
The iOS `keyboard return` path is both slower than nearby keyboard commands and biased toward English keyboard labels.
Current flow:
- `src/core/dispatch.ts` maps `keyboard enter` / `keyboard return` to the XCTest runner command `keyboardReturn`.
- `ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift` calls `tapKeyboardReturnControl(app:)` before trying `app.typeText(XCUIKeyboardKey.return.rawValue)`.
- `tapKeyboardReturnControl` probes English labels in order: `return`, `Return`, `Enter`, `Go`, `Search`, `Next`, `Done`, `Send`, `Join`.
That has two issues:
- Non-English simulator/device languages may expose localized labels, so English label probing is not reliable as the primary path.
- In the Settings search flow, the key is likely `Search`, so the runner pays several failed XCTest element lookups before the successful one. The updated iOS perf harness measured `keyboard return` at about `3833ms` wall median / `3583ms` daemon median, compared with `type` at about `2970ms` wall / `2752ms` daemon.
Perf report with the new harness row:
- `/private/tmp/agent-device-perf-pr643-series-full/perf-ios-2026-06-01T14-44-56-344Z.md`
## Likely Approach
Prefer a language-neutral return-key path before English label probing:
1. If the iOS software keyboard is visible, try `app.typeText(XCUIKeyboardKey.return.rawValue)` first inside the existing Objective-C exception catcher.
2. If that throws, try the existing single text-entry fallback: find the single text input, tap it, then `typeText(.return)`.
3. Keep visible keyboard-control tapping as a fallback, but do not make English label lookup the primary path.
4. If label fallback remains, consider querying visible keyboard buttons/keys once and matching in memory rather than issuing repeated named XCTest queries for each candidate label.
## Acceptance Criteria
- `keyboard return` still works for common iOS return-key variants such as Return, Search, Go, Next, Done, and Send.
- Add or update runner-level/manual validation notes for at least one non-English simulator language so the behavior is not English-label-dependent.
- Run the iOS perf harness and compare `keyboard return` against `type`; the goal is to remove obvious label-probing overhead, not to make keyboard return faster than text entry in every run.
- Preserve the existing unsupported-operation behavior when no keyboard/focused text input can accept return.
## Relevant Code
- `src/core/dispatch.ts` — `handleIosKeyboardCommand` sends `keyboardReturn`.
- `ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift` — runner command dispatch for `.keyboardReturn`.
- `ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift` — `pressKeyboardReturn`, `tapKeyboardReturnControl`, and `singleTextEntryElement`.
Beitragsleitfaden
Erste Schritte
- Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
- Forke das Repository und arbeite in einem Branch.
- Öffne einen Pull Request, der die Issue-Nummer nennt.
Rechercherichtung
Beginne in ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift und verfolge pressKeyboardReturn, tapKeyboardReturnControl und singleTextEntryElement; prüfe dispatch.ts und RunnerTests+CommandExecution.swift auf den Befehlsweg. Führe den bestehenden iOS perf harness aus und eine manuelle Validierung in einem nicht-englischen Simulator durch. Die Arbeit ist abgeschlossen, wenn die Return-Varianten weiterhin funktionieren, nicht unterstützte Operationen unverändert bleiben und der Overhead der Label-Ermittlung reduziert ist.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- ios, swift, typescript
- Bereich
- mobile, performance, testing
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Klar beschrieben
- Anfängerfreundlichkeit
- 55/100