callstack / callstack/agent-device
Treat interaction targets behind the visible keyboard as occluded/off-screen (parity with scroll)
- Dominant language
- TypeScript
- Stars
- 4.6k
- Forks
- 299
- Avg merge
- 10h 17m
- Merged PRs (30d)
- 515
Description
## Purpose
When the software keyboard is up, elements it visually covers still report `hittable: true` with an on-screen `rect`, so `press`/`click`/`fill`/`longpress` tap *through* the keyboard and silently activate whatever the keyboard's own hit-testing routes the point to (or nothing). Keyboard occlusion is a normal, expected iOS/Android pattern — apps are not going to move their tab bar above the IME — so the fix belongs in agent-device: an interaction target whose tap point falls under the visible keyboard should be treated as occluded/off-screen and refused with a typed reason and a recovery hint, exactly the way `scroll` already does.
This came out of dogfooding MaintainX on a physical iPhone (`thymikee-iphone`, v5.39-311017).
## Context / evidence
On the Messages tab of MaintainX, focusing the search field raised the keyboard over the bottom tab bar. The tab-bar nodes stayed in the app snapshot with their layout geometry and were still `hittable`:
```
Button "More" center=(340,822) rect={"x":303,"y":795,"width":74,"height":54}
Button "Messages" center=(274,822) ...
```
The keyboard occupies roughly `y≈540..874`, so the More tab is fully behind it. `press 340 822` (and `press @more`) did **not** navigate; the touch landed on the keyboard. The AX overlay below shows the tab-bar refs drawn *under* the keys:
The annotated overlay (tab-bar refs `@e40`–`@e46` drawn *under* the keys) and a reproduction video are in the dogfood evidence bundle (`bug3-keyboard-covers-tabbar.png`, `bug3-keyboard-tabbar.mp4`); they need a manual web upload to attach here.
(Reproduction video: local `bug3-keyboard-tabbar.mp4` — More tab works with no keyboard, then is dead once the search field is focused. Same class of silent misfire on Android, whose IME is also a separate window.)
## Why the current guards miss it
- `occlusion` (ADR 0011) refuses targets "covered by another visible element" — but occlusion is computed from same-window drawing order in the snapshot tree. The keyboard is a **separate window** (a system surface), so it never appears as a covering sibling of app elements.
- `offscreen` refuses targets whose tap point is outside the **root Application/Window viewport** (`isTapPointInsideViewport` / `isVisibleOnScreen` in `packages/contracts/src/snapshot-visibility.ts`). The covered tab bar is still inside the app window rect, so it passes.
`scroll` already has this exact notion and does the right thing: `SCROLL_KEYBOARD_OCCLUDES_SURFACE_REASON` (`packages/contracts/src/scroll-command.ts`, `packages/contracts/src/scroll-gesture.ts`, golden table `contracts/fixtures/scroll-keyboard-policy.json`) and the documented behavior in `src/commands/interaction/metadata.ts`:
> A visible keyboard shortens the swiped band instead of being dismissed; when too little is left, the command refuses with `scroll_keyboard_occludes_surface`.
Interaction tap paths have no equivalent, which is the asymmetry this issue closes.
## Required behavior
- On press/click/fill/longpress (and their `--settle`/native-ref fast paths), when a software keyboard is visible, compute the keyboard's occluding frame and treat any target whose **tap point (rect center)** falls inside it as occluded/off-screen.
- Fail with a typed reason on that interaction guarantee (mirroring `scroll_keyboard_occludes_surface`), plus a `hint` to dismiss the keyboard first (reference `keyboard dismiss`, and note iOS only dismisses via the keyboard's own control per #1606 / `mechanism: 'dismissKey'` in `packages/contracts/src/keyboard.ts`).
- Do NOT auto-dismiss the keyboard as a side effect — dismissal has semantic consequences (e.g. commits/cancels edit state); refuse and let the caller decide, consistent with the scroll policy.
- Apply to both platforms: iOS software keyboard and Android IME (`focusedPackage`/`inputMethodPackage` path already surfaced in `KeyboardCommandResult`). Coordinate-only taps must be guarded too, since that is how agents hit-tested past the tree here.
- Keyboard-frame source: derive from the already-captured snapshot (`inputView` / keyboard nodes are present in the iOS tree) and/or the runner's keyboard-frame API; on Android from the visible IME / visible-to-user inset. If the keyboard frame is unavailable on a backend, fail **open** (tap proceeds) to avoid regressing current behavior, but record the miss on a typed reason/detail rather than guessing.
## Completion conditions
- A coordinate or `@ref` tap on an app element fully behind the visible keyboard is refused with the new typed reason + dismiss hint, and a follow-up after `keyboard dismiss` succeeds.
- An element only *partially* under the keyboard whose tap point (center) is still above it continues to tap normally (center rule, consistent with `isTapPointInsideViewport`).
- Golden/parity coverage: a keyboard-occlusion fixture under `contracts/fixtures/` (like `scroll-keyboard-policy.json`) and an entry for the new path/guarantee stance in `packages/contracts/src/interaction-guarantees.ts` for `occlusion` and/or `offscreen`.
- Versioned CLI help / metadata updated to name the new refusal reason.
## API / CLI shape
```
# before: silently no-ops (tap hits the keyboard)
agent-device press @more --settle
# after:
Error (COMMAND_FAILED): Target @more is behind the visible keyboard and cannot be tapped
Hint: Run `agent-device keyboard dismiss` (iOS taps the keyboard's own dismiss control) and retry.
details: { reason: 'tap_keyboard_occludes_target', keyboardFrame: {...}, targetRect: {...} }
```
## Dependencies / open questions
- Needs a reliable keyboard-frame provider on the iOS runner (incl. physical devices) and Android; confirm both are exposed at snapshot time before choosing snapshot-derived vs. separate API.
- Decide whether this lands as a new guarantee cell or extends the existing `offscreen` enforcement set; if it can be computed purely from the captured snapshot node, it likely belongs in the shared runtime preflight (`preflightNativeRefInteraction`, #1081) so the fast path can't silently succeed.
Contributor guide
Assessment
This issue has not been assessed yet.