callstack / callstack/agent-device

Treat interaction targets behind the visible keyboard as occluded/off-screen (parity with scroll)

Chiusa
#2,589 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
needs-triage
Lingua principale
TypeScript
Stelle
4.6k
Fork
299
Merge medio
10h 42m
PR unite (30g)
493

Descrizione

## Purpose

When the software keyboard is up, elements it visually covers still report `hittable: true` with an on-screen `rect`, so `press`/`click`/`fill`/`longpress` tap *through* the keyboard and silently activate whatever the keyboard's own hit-testing routes the point to (or nothing). Keyboard occlusion is a normal, expected iOS/Android pattern — apps are not going to move their tab bar above the IME — so the fix belongs in agent-device: an interaction target whose tap point falls under the visible keyboard should be treated as occluded/off-screen and refused with a typed reason and a recovery hint, exactly the way `scroll` already does.

This came out of dogfooding MaintainX on a physical iPhone (`thymikee-iphone`, v5.39-311017).

## Context / evidence

On the Messages tab of MaintainX, focusing the search field raised the keyboard over the bottom tab bar. The tab-bar nodes stayed in the app snapshot with their layout geometry and were still `hittable`:

```
Button "More" center=(340,822) rect={"x":303,"y":795,"width":74,"height":54}
Button "Messages" center=(274,822) ...
```

The keyboard occupies roughly `y≈540..874`, so the More tab is fully behind it. `press 340 822` (and `press @more`) did **not** navigate; the touch landed on the keyboard. The AX overlay below shows the tab-bar refs drawn *under* the keys:

The annotated overlay (tab-bar refs `@e40`–`@e46` drawn *under* the keys) and a reproduction video are in the dogfood evidence bundle (`bug3-keyboard-covers-tabbar.png`, `bug3-keyboard-tabbar.mp4`); they need a manual web upload to attach here.

(Reproduction video: local `bug3-keyboard-tabbar.mp4` — More tab works with no keyboard, then is dead once the search field is focused. Same class of silent misfire on Android, whose IME is also a separate window.)

## Why the current guards miss it

- `occlusion` (ADR 0011) refuses targets "covered by another visible element" — but occlusion is computed from same-window drawing order in the snapshot tree. The keyboard is a **separate window** (a system surface), so it never appears as a covering sibling of app elements.
- `offscreen` refuses targets whose tap point is outside the **root Application/Window viewport** (`isTapPointInsideViewport` / `isVisibleOnScreen` in `packages/contracts/src/snapshot-visibility.ts`). The covered tab bar is still inside the app window rect, so it passes.

`scroll` already has this exact notion and does the right thing: `SCROLL_KEYBOARD_OCCLUDES_SURFACE_REASON` (`packages/contracts/src/scroll-command.ts`, `packages/contracts/src/scroll-gesture.ts`, golden table `contracts/fixtures/scroll-keyboard-policy.json`) and the documented behavior in `src/commands/interaction/metadata.ts`:

> A visible keyboard shortens the swiped band instead of being dismissed; when too little is left, the command refuses with `scroll_keyboard_occludes_surface`.

Interaction tap paths have no equivalent, which is the asymmetry this issue closes.

## Required behavior

- On press/click/fill/longpress (and their `--settle`/native-ref fast paths), when a software keyboard is visible, compute the keyboard's occluding frame and treat any target whose **tap point (rect center)** falls inside it as occluded/off-screen.
- Fail with a typed reason on that interaction guarantee (mirroring `scroll_keyboard_occludes_surface`), plus a `hint` to dismiss the keyboard first (reference `keyboard dismiss`, and note iOS only dismisses via the keyboard's own control per #1606 / `mechanism: 'dismissKey'` in `packages/contracts/src/keyboard.ts`).
- Do NOT auto-dismiss the keyboard as a side effect — dismissal has semantic consequences (e.g. commits/cancels edit state); refuse and let the caller decide, consistent with the scroll policy.
- Apply to both platforms: iOS software keyboard and Android IME (`focusedPackage`/`inputMethodPackage` path already surfaced in `KeyboardCommandResult`). Coordinate-only taps must be guarded too, since that is how agents hit-tested past the tree here.
- Keyboard-frame source: derive from the already-captured snapshot (`inputView` / keyboard nodes are present in the iOS tree) and/or the runner's keyboard-frame API; on Android from the visible IME / visible-to-user inset. If the keyboard frame is unavailable on a backend, fail **open** (tap proceeds) to avoid regressing current behavior, but record the miss on a typed reason/detail rather than guessing.

## Completion conditions

- A coordinate or `@ref` tap on an app element fully behind the visible keyboard is refused with the new typed reason + dismiss hint, and a follow-up after `keyboard dismiss` succeeds.
- An element only *partially* under the keyboard whose tap point (center) is still above it continues to tap normally (center rule, consistent with `isTapPointInsideViewport`).
- Golden/parity coverage: a keyboard-occlusion fixture under `contracts/fixtures/` (like `scroll-keyboard-policy.json`) and an entry for the new path/guarantee stance in `packages/contracts/src/interaction-guarantees.ts` for `occlusion` and/or `offscreen`.
- Versioned CLI help / metadata updated to name the new refusal reason.

## API / CLI shape

```
# before: silently no-ops (tap hits the keyboard)
agent-device press @more --settle

# after:
Error (COMMAND_FAILED): Target @more is behind the visible keyboard and cannot be tapped
Hint: Run `agent-device keyboard dismiss` (iOS taps the keyboard's own dismiss control) and retry.
details: { reason: 'tap_keyboard_occludes_target', keyboardFrame: {...}, targetRect: {...} }
```

## Dependencies / open questions

- Needs a reliable keyboard-frame provider on the iOS runner (incl. physical devices) and Android; confirm both are exposed at snapshot time before choosing snapshot-derived vs. separate API.
- Decide whether this lands as a new guarantee cell or extends the existing `offscreen` enforcement set; if it can be computed purely from the captured snapshot node, it likely belongs in the shared runtime preflight (`preflightNativeRefInteraction`, #1081) so the fast path can't silently succeed.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia leggendo packages/contracts/src/snapshot-visibility.ts, scroll-command.ts, scroll-gesture.ts e src/commands/interaction/metadata.ts, quindi esamina il fixture scroll-keyboard-policy.json e interaction-guarantees.ts. Traccia il preflight di interazione condiviso, incluso il percorso preflightNativeRefInteraction menzionato nell’issue, attraverso le fonti dei keyboard-frame di iOS e Android. Il lavoro è completato quando passano i controlli di coordinate/ref, center-rule, dismiss-follow-up, fail-open, golden fixture e CLI metadata.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
android, ios, typescript
Ambito
cli, mobile, testing
Tipo di issue
Bug
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Attiva
Chiarezza
Abbastanza chiara
Idoneità per principianti
42/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.