web-infra-dev / web-infra-dev/midscene
[Bug]: Android Studio Recorder captures stale ADB screenshots instead of the current scrcpy preview frame
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 14.9k
- Forks
- 1.2k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 96
Description
Version
Midscene Studio: [fill packaged version, if available]
Source commit: 69a7bcadfef3ad0c7a5cc917d613c26f3f24245f
Details
When recording an Android workflow in Midscene Studio, the user interacts
with the live scrcpy preview. However, the Recorder screenshot associated
with the interaction does not come from the currently displayed scrcpy frame.
The Studio Android runtime currently creates two independent capture paths:
- A standalone
ScrcpyServersidecar provides the live Studio preview. - The Recorder captures screenshots through
agent.interface.screenshotBase64().
The Android Agent is created using new AndroidDevice(deviceId) without
enabling scrcpyConfig. Therefore, Recorder screenshots use the normal ADB
takeScreenshot / screencap path instead of the scrcpy stream displayed
in Studio.
For the screenshot before an interaction, Recorder first tries the last
MJPEG frame and then falls back to _studioPreviewRecorderLastScreenshot.
When Android scrcpy preview is active, the renderer does not consume the
Playground MJPEG endpoint. As a result, this fallback may be the screenshot
captured when recording started or after the previous interaction.
This produces a mismatch:
- The recorded coordinates refer to the current scrcpy frame visible to the user.
- The screenshot used by Recorder and
describeElementAtPointmay represent
an older UI state. - The generated YAML/Markdown script can describe the wrong target or fail
to replay.
Raw Tap/Input events are generally still recorded. The primary impact is
incorrect screenshots, semantic element descriptions, and generated scripts.
This is related to #2712, but it is specifically caused by Studio's Android
Recorder and scrcpy preview using separate frame sources.
Reproduce link
none
Reproduce Steps
- Open Midscene Studio and connect an Android target.
- Confirm that the live scrcpy preview is connected.
- Open an Android screen that automatically changes from UI state A to UI
state B without a Studio interaction. - Start Recorder while state A is visible.
- Wait until the scrcpy preview changes to state B.
- Tap an element that exists in state B.
- Stop recording.
- Inspect the screenshot stored for the Tap event and the generated element
description/script.
Actual result:
- The Tap coordinates correspond to state B.
- The Recorder screenshot may still show state A—the screenshot captured at
Recorder startup or after the previous interaction. - AI element description or generated script can reference the wrong element.
Expected result:
- Recorder should use the current scrcpy frame visible when the interaction
begins. - Screenshot dimensions and coordinates should use the same stream coordinate
space. - If no valid scrcpy frame is available, Recorder may fall back to
screenshotBase64()/ADB.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the Studio Android runtime from new AndroidDevice(deviceId), ScrcpyServer, screenshotBase64(), and _studioPreviewRecorderLastScreenshot. Compare the Recorder's pre-interaction screenshot path with the live scrcpy frame and verify coordinate dimensions. Done means Recorder uses the current scrcpy frame when available and falls back to ADB only when no valid frame exists.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, typescript
- Domain
- mobile-dev, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100