callstack / callstack/agent-device
Gesture viewport should expose content-safe bounds (app window minus system bars) so scroll planning clamps to real geometry
- Dominant language
- TypeScript
- Stars
- 4.6k
- Forks
- 299
- Avg merge
- 10h 14m
- Merged PRs (30d)
- 537
Description
Follow-up from #1820 (#1781 A1), asked for in its review.
## Today
`buildScrollGesturePlan` (`packages/contracts/src/scroll-gesture.ts`) keeps scroll gestures out of the outer `DEFAULT_EDGE_PADDING_FRACTION` = 10 % of each axis. That fraction is a proxy for the real invariant — *a scroll must never touch down inside a system bar* — tuned to exceed the tallest known bar (Pixel 7 cutout status bar 136/2400 = 5.7 %, iPhone Dynamic Island 59/852 = 6.9 %). It lives in a device-agnostic pure planner because the viewport it receives is the raw application window, which on edge-to-edge apps (API 35+, every system app, iOS) starts at y=0 and includes the bars. #1820 raised it from 5 % after `scroll up 3` on a Pixel 7-geometry emulator touched down at y=120, inside the bar, and pulled the notification shade instead of scrolling.
Costs of the proxy: every user pays 10 points of max single-gesture travel (90 % → 80 % for `amount ≥ 0.8`) whether or not their device has a cutout, and a device with a taller bar than the tuned constant reintroduces the shade pull silently.
## Proposed
The layer that knows where the bars are should say so:
- **Android** — `GestureViewportReader.read()` (`android/snapshot-helper/.../GestureViewportReader.java`) already iterates `UiAutomation.getWindows()` and keeps only `TYPE_APPLICATION`; the `StatusBar` / `NavigationBar` system windows are right there to subtract (or read the status/navigation-bar insets). Expose *content-safe bounds* alongside the raw window bounds in the `viewport` action (helper protocol/API version bump).
- **Apple** — the runner has safe-area insets; report the same content-safe rect in the resolved scroll frame.
- **Planner** — clamp scroll travel to the content-safe rect and return `DEFAULT_EDGE_PADDING_FRACTION` to a small generic margin (the original 5 % or less). Keep the raw window rect for absolute `swipe`/`gesture` validation so an agent can still deliberately swipe down from the status bar.
- Parity table `contracts/fixtures/scroll-gesture.json` (added in #1820) is where the new vectors go — both the TS and Swift suites iterate it.
## Also on this issue: suite discovery order
The masking in #1781 A1 came from `discoverReplaySourcePaths` (`src/daemon/replay-source-discovery.ts`) enumerating directory inputs in native readdir order (`03,05,06,…,04` on the ubuntu runner vs `01…06` locally). The header documents this as deliberate Maestro parity, and it is accurate — Maestro's `WorkspaceExecutionPlanner` uses `Files.walk` and names the result `unsortedFlowFiles`, no sort. Sorting directory inputs would make results reproducible across machines but is a deliberate departure from that parity; decide it here rather than silently. Order-sensitive tests currently pass explicit positionals (see `session-test-suite.test.ts`).
Contributor guide
Research direction
Start with packages/contracts/src/scroll-gesture.ts and GestureViewportReader.java, then trace the viewport protocol and Apple scroll-frame handling. Update contracts/fixtures/scroll-gesture.json and run the TypeScript and Swift suites to verify content-safe clamping while raw gesture bounds remain available. Also review src/daemon/replay-source-discovery.ts and session-test-suite.test.ts to decide whether directory discovery should preserve Maestro's unsorted behavior or become reproducible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, ios, java, swift, typescript
- Domain
- api, mobile, testing, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100