callstack / callstack/agent-device

Gesture viewport should expose content-safe bounds (app window minus system bars) so scroll planning clamps to real geometry

Open
#1,821 1 comment 0 reactions 0 assignees View on GitHub
needs-info
Dominant language
TypeScript
Stars
4.6k
Forks
299
Avg merge
10h 14m
Merged PRs (30d)
537

Description

Follow-up from #1820 (#1781 A1), asked for in its review.

## Today
`buildScrollGesturePlan` (`packages/contracts/src/scroll-gesture.ts`) keeps scroll gestures out of the outer `DEFAULT_EDGE_PADDING_FRACTION` = 10 % of each axis. That fraction is a proxy for the real invariant — *a scroll must never touch down inside a system bar* — tuned to exceed the tallest known bar (Pixel 7 cutout status bar 136/2400 = 5.7 %, iPhone Dynamic Island 59/852 = 6.9 %). It lives in a device-agnostic pure planner because the viewport it receives is the raw application window, which on edge-to-edge apps (API 35+, every system app, iOS) starts at y=0 and includes the bars. #1820 raised it from 5 % after `scroll up 3` on a Pixel 7-geometry emulator touched down at y=120, inside the bar, and pulled the notification shade instead of scrolling.

Costs of the proxy: every user pays 10 points of max single-gesture travel (90 % → 80 % for `amount ≥ 0.8`) whether or not their device has a cutout, and a device with a taller bar than the tuned constant reintroduces the shade pull silently.

## Proposed
The layer that knows where the bars are should say so:
- **Android** — `GestureViewportReader.read()` (`android/snapshot-helper/.../GestureViewportReader.java`) already iterates `UiAutomation.getWindows()` and keeps only `TYPE_APPLICATION`; the `StatusBar` / `NavigationBar` system windows are right there to subtract (or read the status/navigation-bar insets). Expose *content-safe bounds* alongside the raw window bounds in the `viewport` action (helper protocol/API version bump).
- **Apple** — the runner has safe-area insets; report the same content-safe rect in the resolved scroll frame.
- **Planner** — clamp scroll travel to the content-safe rect and return `DEFAULT_EDGE_PADDING_FRACTION` to a small generic margin (the original 5 % or less). Keep the raw window rect for absolute `swipe`/`gesture` validation so an agent can still deliberately swipe down from the status bar.
- Parity table `contracts/fixtures/scroll-gesture.json` (added in #1820) is where the new vectors go — both the TS and Swift suites iterate it.

## Also on this issue: suite discovery order
The masking in #1781 A1 came from `discoverReplaySourcePaths` (`src/daemon/replay-source-discovery.ts`) enumerating directory inputs in native readdir order (`03,05,06,…,04` on the ubuntu runner vs `01…06` locally). The header documents this as deliberate Maestro parity, and it is accurate — Maestro's `WorkspaceExecutionPlanner` uses `Files.walk` and names the result `unsortedFlowFiles`, no sort. Sorting directory inputs would make results reproducible across machines but is a deliberate departure from that parity; decide it here rather than silently. Order-sensitive tests currently pass explicit positionals (see `session-test-suite.test.ts`).

Contributor guide

Open the contributing guide

Research direction

Start with packages/contracts/src/scroll-gesture.ts and GestureViewportReader.java, then trace the viewport protocol and Apple scroll-frame handling. Update contracts/fixtures/scroll-gesture.json and run the TypeScript and Swift suites to verify content-safe clamping while raw gesture bounds remain available. Also review src/daemon/replay-source-discovery.ts and session-test-suite.test.ts to decide whether directory discovery should preserve Maestro's unsorted behavior or become reproducible.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, ios, java, swift, typescript
Domain
api, mobile, testing, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.