callstack / callstack/agent-device

Improve iOS keyboard return reliability and perf without English label probing

Aperta
#646 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
enhancement ready-for-agent
Lingua principale
TypeScript
Stelle
4.6k
Fork
299
Merge medio
10h 42m
PR unite (30g)
493

Descrizione

> *This was generated by AI during triage.*

## Problem

The iOS `keyboard return` path is both slower than nearby keyboard commands and biased toward English keyboard labels.

Current flow:

- `src/core/dispatch.ts` maps `keyboard enter` / `keyboard return` to the XCTest runner command `keyboardReturn`.
- `ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift` calls `tapKeyboardReturnControl(app:)` before trying `app.typeText(XCUIKeyboardKey.return.rawValue)`.
- `tapKeyboardReturnControl` probes English labels in order: `return`, `Return`, `Enter`, `Go`, `Search`, `Next`, `Done`, `Send`, `Join`.

That has two issues:

- Non-English simulator/device languages may expose localized labels, so English label probing is not reliable as the primary path.
- In the Settings search flow, the key is likely `Search`, so the runner pays several failed XCTest element lookups before the successful one. The updated iOS perf harness measured `keyboard return` at about `3833ms` wall median / `3583ms` daemon median, compared with `type` at about `2970ms` wall / `2752ms` daemon.

Perf report with the new harness row:

- `/private/tmp/agent-device-perf-pr643-series-full/perf-ios-2026-06-01T14-44-56-344Z.md`

## Likely Approach

Prefer a language-neutral return-key path before English label probing:

1. If the iOS software keyboard is visible, try `app.typeText(XCUIKeyboardKey.return.rawValue)` first inside the existing Objective-C exception catcher.
2. If that throws, try the existing single text-entry fallback: find the single text input, tap it, then `typeText(.return)`.
3. Keep visible keyboard-control tapping as a fallback, but do not make English label lookup the primary path.
4. If label fallback remains, consider querying visible keyboard buttons/keys once and matching in memory rather than issuing repeated named XCTest queries for each candidate label.

## Acceptance Criteria

- `keyboard return` still works for common iOS return-key variants such as Return, Search, Go, Next, Done, and Send.
- Add or update runner-level/manual validation notes for at least one non-English simulator language so the behavior is not English-label-dependent.
- Run the iOS perf harness and compare `keyboard return` against `type`; the goal is to remove obvious label-probing overhead, not to make keyboard return faster than text entry in every run.
- Preserve the existing unsupported-operation behavior when no keyboard/focused text input can accept return.

## Relevant Code

- `src/core/dispatch.ts` — `handleIosKeyboardCommand` sends `keyboardReturn`.
- `ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift` — runner command dispatch for `.keyboardReturn`.
- `ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift` — `pressKeyboardReturn`, `tapKeyboardReturnControl`, and `singleTextEntryElement`.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia da ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Interaction.swift, seguendo pressKeyboardReturn, tapKeyboardReturnControl e singleTextEntryElement; controlla dispatch.ts e RunnerTests+CommandExecution.swift per il percorso del comando. Esegui l’iOS perf harness esistente e la validazione manuale in un simulatore non in inglese. Il lavoro è completato quando le varianti di Return continuano a funzionare, le operazioni non supportate rimangono invariate e l’overhead del rilevamento delle label è ridotto.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
ios, swift, typescript
Ambito
mobile, performance, testing
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Specificata chiaramente
Idoneità per principianti
55/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.