callstack / callstack/agent-device

Limrun iOS: Flutter text entry fails with “No focused accessibility element found” after tapping an editable field

Open
#2,578 2 comments 0 reactions 0 assignees View on GitHub
needs-info
Dominant language
TypeScript
Stars
4.6k
Forks
299
Avg merge
10h 17m
Merged PRs (30d)
515

Description

## Reproduction and observed behavior

Using agent-device 0.21.0 against a Flutter login screen on a Limrun iOS cloud simulator, snapshot exposes Email and Password as editable text fields. Filling Email, or tapping the field and using type, fails with:

```text
No focused accessibility element found — Unexpected internal error. Retry with --debug and report the diagnostics log if it persists.
```

The first attempt may display the iOS slide-to-type tutorial. Dismissing it and taking a fresh snapshot does not resolve subsequent typing failures. Repeated ref/coordinate focus and type attempts still fail.

The same Flutter app accepts input locally. A React Native app tested on Limrun does not exhibit the failure. This suggests a compatibility problem in this path, not proof of a particular Flutter engine defect. Exact Flutter/iOS versions and a standalone public reproduction are still needed.

## Current implementation

At e3880f10c826fea373cc48153996b372ef39136e, `packages/provider-limrun/src/ios.ts` sends both `type` and `fill` through `client.typeText`; `fill` first taps the supplied coordinates. The repository resolves `@limrun/api` 0.24.5, whose `typeText` signature has no focus-check option.

## Confirmed downstream workaround

We patched the bundled iOS client’s `typeText` request sender in agent-device 0.21.0:

1. Send the normal request first.
2. Only on the exact server error `No focused accessibility element found` (without the CLI’s appended diagnostic hint), retry once with the same `text` and `pressEnter` and `requireFocus: false`.
3. Propagate all other failures and any failure of the recovery request; never replay on a timeout or unknown failure.

We confirmed that this restores typing in the affected Flutter app. The workaround stays inside agent-device; the QA application does not call Limrun directly. This is a tested downstream workaround, not validation of a patch against current main.

## Proposed upstream direction

Use a supported SDK exposing `typeText(text, pressEnter, { requireFocus: false })`, and agree on a typed pre-input focus rejection or an explicit focus-bypass policy. Current upstream guidance forbids branching on error-message text; the temporary workaround should not be copied unchanged.

Do not bypass the focus check globally or retry after potentially partial writes. Preserve type/fill semantics and verify the intended field’s value. Keep keyboard tutorial handling separate.

Validation should include normal input, focus-check rejection, timeout/no replay, unsuccessful recovery, delayed character entry, existing-value fill, password fields, and live Flutter/React Native verification on Limrun. An SDK upgrade must also be validated against the provider’s existing lifecycle and recording API usage.

Would maintainers prefer a typed SDK error prerequisite or an explicit adapter option for input after a resolved field tap?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.