web-infra-dev / web-infra-dev/midscene

[Bug]: Android Studio Recorder captures stale ADB screenshots instead of the current scrcpy preview frame

Open
#2,911 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
14.9k
Forks
1.2k
Avg merge
1d 14h
Merged PRs (30d)
96

Description

Version
Midscene Studio: [fill packaged version, if available]
Source commit: 69a7bcadfef3ad0c7a5cc917d613c26f3f24245f
Details

When recording an Android workflow in Midscene Studio, the user interacts
with the live scrcpy preview. However, the Recorder screenshot associated
with the interaction does not come from the currently displayed scrcpy frame.

The Studio Android runtime currently creates two independent capture paths:

  1. A standalone ScrcpyServer sidecar provides the live Studio preview.
  2. The Recorder captures screenshots through
    agent.interface.screenshotBase64().

The Android Agent is created using new AndroidDevice(deviceId) without
enabling scrcpyConfig. Therefore, Recorder screenshots use the normal ADB
takeScreenshot / screencap path instead of the scrcpy stream displayed
in Studio.

For the screenshot before an interaction, Recorder first tries the last
MJPEG frame and then falls back to _studioPreviewRecorderLastScreenshot.
When Android scrcpy preview is active, the renderer does not consume the
Playground MJPEG endpoint. As a result, this fallback may be the screenshot
captured when recording started or after the previous interaction.

This produces a mismatch:

  • The recorded coordinates refer to the current scrcpy frame visible to the user.
  • The screenshot used by Recorder and describeElementAtPoint may represent
    an older UI state.
  • The generated YAML/Markdown script can describe the wrong target or fail
    to replay.

Raw Tap/Input events are generally still recorded. The primary impact is
incorrect screenshots, semantic element descriptions, and generated scripts.

This is related to #2712, but it is specifically caused by Studio's Android
Recorder and scrcpy preview using separate frame sources.

Reproduce link

none

Reproduce Steps
  1. Open Midscene Studio and connect an Android target.
  2. Confirm that the live scrcpy preview is connected.
  3. Open an Android screen that automatically changes from UI state A to UI
    state B without a Studio interaction.
  4. Start Recorder while state A is visible.
  5. Wait until the scrcpy preview changes to state B.
  6. Tap an element that exists in state B.
  7. Stop recording.
  8. Inspect the screenshot stored for the Tap event and the generated element
    description/script.

Actual result:

  • The Tap coordinates correspond to state B.
  • The Recorder screenshot may still show state A—the screenshot captured at
    Recorder startup or after the previous interaction.
  • AI element description or generated script can reference the wrong element.

Expected result:

  • Recorder should use the current scrcpy frame visible when the interaction
    begins.
  • Screenshot dimensions and coordinates should use the same stream coordinate
    space.
  • If no valid scrcpy frame is available, Recorder may fall back to
    screenshotBase64()/ADB.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Studio Android runtime from new AndroidDevice(deviceId), ScrcpyServer, screenshotBase64(), and _studioPreviewRecorderLastScreenshot. Compare the Recorder's pre-interaction screenshot path with the live scrcpy frame and verify coordinate dimensions. Done means Recorder uses the current scrcpy frame when available and falls back to ADB only when no valid frame exists.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, typescript
Domain
mobile-dev, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.