Recording touch overlay exports at ~220x480 or all-black whenever touches are drawn

Open
#2,707 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
56/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
ios, typescript

Research direction

Start at the record transport and compare the default overlay path with --hide-touches, using the raw capture path as the size reference. Add a regression test for track-size equality with a scripted tap and ensure overlay failure is reported rather than producing black output. Correct the claims in docs/adr/0025-foldable-apple-panels.md and docs/agents/device-verification.md, then re-measure the listed non-rot90 and iPhone Duo cases.

Written by the indexing model from the issue text.

Description

needs-triage

Purpose

record with the default touch overlay produces an unusable export as soon as the recording contains an interaction: the track collapses to roughly 220x480 regardless of the device's native capture size, and on a long recording it comes out entirely black. Touch indicators are therefore unavailable for exactly the recordings that need them, and the caller gets exit code 0 either way.

This is the recording transport, not the display-selection path. The raw capture honors --display correctly since #2703.

Observed

iPhone 17 simulator 0E0FCBDC-88E2-4CB6-A1DA-E148F77743F4 (iOS 27.0, non-rot90 panel), Xcode 27.1 beta, agent-device at 057dfb7c1. Mean luma measured over sampled frames; content changes across frames are marked where they occur.

Overlay Interactions during capture Exported size Result
--hide-touches none 1206x2622 correct
default none 1206x2622 correct
default two tab taps 220x480 bright, content changes present
--hide-touches two tab taps 1206x2622 correct, content changes present
default taps, scroll, 97s 480x220 meanLuma=0.00 every frame

Two things follow from the matrix. The trigger is the overlay drawing touch events, not the interactions themselves — --hide-touches stays full size while recording the same taps. And it is not orientation-dependent: the failing device here has a normal portrait panel.

The overlay does draw something, since the small export tracks the screen. Only the geometry and, in the long case, the frame content are lost.

Required behavior

  • A recording with touches exports at the same track size as the same recording without touches.
  • A recording never exports all-black. If the overlay compositor cannot produce frames, fail the command or drop the overlay and say so in the result, rather than returning a black file with exit 0.
  • The choice to drop the overlay must be reported on the response, not inferred from pixel dimensions by whoever notices later.

Completion conditions

  • Table above re-measured on a non-rot90 target and on the iPhone Duo inner panel, with the overlay rows matching the --hide-touches rows for size and non-blackness.
  • A regression test asserts exported track size equals the raw path's for a scripted tap, since "file exists and is non-zero bytes" would have passed all four broken rows above.
  • docs/adr/0025-foldable-apple-panels.md and the --hide-touches note in docs/agents/device-verification.md currently state that the defect is specific to rot90 inner-panel tracks and that the same path works on non-rotated panels. That claim is wrong and was written before this matrix existed; both need correcting as part of this work.

Dependencies

None. Reproducible on main.

Dominant language
TypeScript
Stars
4.7k
Forks
304
Avg merge
11h 46m
Merged PRs (30d)
471

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from callstack/agent-device

All issues in callstack/agent-device

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.