callstack / callstack/agent-device

scroll keyboard clip follow-ups: double keyboard probe on iOS, tap/drag width-fraction gate, horizontal-scroll refusal

Open
#2,554 0 comments 0 reactions 0 assignees View on GitHub
backlog enhancement
Dominant language
TypeScript
Stars
4.6k
Forks
299
Avg merge
10h 17m
Merged PRs (30d)
515

Description

## Context

The #2500 stack (#2537 → #2503 → #2514) clips a directional scroll's swipe above the on-screen keyboard and refuses with `scroll_keyboard_occludes_surface` when too little surface is left. Three things were found in review and left out of the stack on purpose. None blocks merge; each is its own small change.

## 1. iOS scroll probes the keyboard twice per command

`resolvedScrollViewport` reads `visibleKeyboardFrame` to clip the band. The dispatch then goes through `executeSynthesizedDragGesture` → `axFreeSynthesizedDragPlan` → `keyboardAvoidingSynthesizedDragPoints` (`RunnerTests+Interaction.swift:912`), which reads `visibleKeyboardFrame` again and shifts the drag if it intersects the keyboard. For a scroll the second pass is a no-op by construction: the band already ends `accessoryAllowance` above `keyboardMinY`, so the planned points never intersect. Each probe is a live `app.keyboards.firstMatch` query.

Before the stack, the first scroll of a session ran zero probes (policy `.whenAccessibilityHealthy` on `.unknown`); it now runs two. Options:

- carry the frame the clip measured into the context so the drag pass reuses it, or
- give the scroll dispatch a context with `keyboardPolicy: .never` at `RunnerTests+CommandExecution.swift:1995`, since the clip already accounted for the keyboard.

Measure the saved time on a large tree (Bluesky or the deep-tree bench) before choosing; the `AGENT_DEVICE_RUNNER_SYNTHESIZED_GESTURE_POLICY` log line must still make sense for scroll.

## 2. Tap/drag reference frame keeps the width-fraction gate

`frameAvoidingKeyboard` (`RunnerTests+Interaction.swift:780`) still gates on `intersection.width / frame.width >= 0.5` (line 790) and fails open under 25% safe height. That is the same defect #2503 removed from scroll: a floating keyboard centred at 38% of the width sits in the path and is ignored. It shapes tap and drag reference frames, so fixing it changes the tap family and needs its own device evidence. `keyboardAvoidingDragPoints` (`:672`) also keeps a local `padding = 12` that names the same fact as `SCROLL_KEYBOARD_ACCESSORY_ALLOWANCE`.

Decide whether the tap family should share `clipScrollViewportAboveKeyboard` (and its golden table) or stay separate with its own fail-open policy. Do not silently switch tap to fail-closed.

## 3. Horizontal scrolls refuse under the vertical floor

`clipScrollViewportAboveKeyboard` (`packages/contracts/src/scroll-gesture.ts:412`) applies the 15% floor regardless of direction. A `scroll left`/`right` runs along the horizontal centre line of the clipped band, so a thin band can still carry it; today it is refused. Rare and conservative. If it matters, pass the direction into the clip and only apply the floor to vertical scrolls, adding rows to `contracts/fixtures/scroll-keyboard-policy.json` so the Swift twin stays in step.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.