callstack / callstack/agent-device
Improve iOS keyboard return reliability and perf without English label probing
- Lingua principale
- TypeScript
- Stelle
- 4.6k
- Fork
- 299
- Merge medio
- 10h 42m
- PR unite (30g)
- 493
Descrizione
> *This was generated by AI during triage.*
## Problem
The iOS `keyboard return` path is both slower than nearby keyboard commands and biased toward English keyboard labels.
Current flow:
- `src/core/dispatch.ts` maps `keyboard enter` / `keyboard return` to the XCTest runner command `keyboardReturn`.
- `ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift` calls `tapKeyboardReturnControl(app:)` before trying `app.typeText(XCUIKeyboardKey.return.rawValue)`.
- `tapKeyboardReturnControl` probes English labels in order: `return`, `Return`, `Enter`, `Go`, `Search`, `Next`, `Done`, `Send`, `Join`.
That has two issues:
- Non-English simulator/device languages may expose localized labels, so English label probing is not reliable as the primary path.
- In the Settings search flow, the key is likely `Search`, so the runner pays several failed XCTest element lookups before the successful one. The updated iOS perf harness measured `keyboard return` at about `3833ms` wall median / `3583ms` daemon median, compared with `type` at about `2970ms` wall / `2752ms` daemon.
Perf report with the new harness row:
- `/private/tmp/agent-device-perf-pr643-series-full/perf-ios-2026-06-01T14-44-56-344Z.md`
## Likely Approach
Prefer a language-neutral return-key path before English label probing:
1. If the iOS software keyboard is visible, try `app.typeText(XCUIKeyboardKey.return.rawValue)` first inside the existing Objective-C exception catcher.
2. If that throws, try the existing single text-entry fallback: find the single text input, tap it, then `typeText(.return)`.
3. Keep visible keyboard-control tapping as a fallback, but do not make English label lookup the primary path.
4. If label fallback remains, consider querying visible keyboard buttons/keys once and matching in memory rather than issuing repeated named XCTest queries for each candidate label.
## Acceptance Criteria
- `keyboard return` still works for common iOS return-key variants such as Return, Search, Go, Next, Done, and Send.
- Add or update runner-level/manual validation notes for at least one non-English simulator language so the behavior is not English-label-dependent.
- Run the iOS perf harness and compare `keyboard return` against `type`; the goal is to remove obvious label-probing overhead, not to make keyboard return faster than text entry in every run.
- Preserve the existing unsupported-operation behavior when no keyboard/focused text input can accept return.
## Relevant Code
- `src/core/dispatch.ts` — `handleIosKeyboardCommand` sends `keyboardReturn`.
- `ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift` — runner command dispatch for `.keyboardReturn`.
- `ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift` — `pressKeyboardReturn`, `tapKeyboardReturnControl`, and `singleTextEntryElement`.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia da ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift, seguendo pressKeyboardReturn, tapKeyboardReturnControl e singleTextEntryElement; controlla dispatch.ts e RunnerTests+CommandExecution.swift per il percorso del comando. Esegui l’iOS perf harness esistente e la validazione manuale in un simulatore non in inglese. Il lavoro è completato quando le varianti di Return continuano a funzionare, le operazioni non supportate rimangono invariate e l’overhead del rilevamento delle label è ridotto.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- ios, swift, typescript
- Ambito
- mobile, performance, testing
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Specificata chiaramente
- Idoneità per principianti
- 55/100