[Feature]: Add bounded-latency flow control to the Android Studio scrcpy preview

What problem does this feature solve?

Version

• Midscene: 1.11.0 / latest main
• Source commit checked: dfc5a5f6ce41abb4f01d4be8c4d998ffa2b5a8c1
• Runtime: Midscene Studio on Windows
• Android device: HONOR BRC-AN00
• Preview profile: 1600px maximum size, 8 Mbps H.264

Feature request

It would be helpful for the Android Studio scrcpy preview to provide bounded-latency flow control between video production and rendering.

The current preview pipeline is:

Android screen
  -> scrcpy / MediaCodec encoder
  -> ADB video stream
  -> android-playground sidecar
  -> Socket.IO video-data events
  -> Electron renderer
  -> ReadableStream
  -> WebCodecs VideoDecoder
  -> VideoFrame renderer
  -> preview canvas

During continuous scrolling, animation, or an AI use case that adds renderer load, the producer can generate frames faster than the renderer can decode and draw them.

In this situation, the preview may gradually fall behind the physical device. After transport congestion or frame loss, it may also take some time to return to the latest device state.

Motivation

A live preview generally benefits more from freshness than from displaying every intermediate frame.

When the consumer cannot keep up, continuing to decode old compressed frames increases latency. For a Studio preview, it would be preferable to selectively discard obsolete frames and resume from a recent keyframe.

Additional instrumentation from an affected run showed:

• Sidecar stream-read gaps remained below approximately 213 ms.
• Renderer-observed packet age reached approximately 5.268 seconds.
• The renderer received 25,634 packets but drew 16,586 frames.
• 7,082 sequence numbers were not observed by the renderer.
• ReadableStream.desiredSize remained at 4.

These observations suggest that device-to-sidecar reading can remain close to real time while work accumulates later in the renderer pipeline.

Current capability gap

Producer rate

The Studio preview requests a 1600px, 8 Mbps stream, but does not currently define an explicit preview frame-rate limit.

A device can therefore produce frames at its display refresh rate even when the preview consumer does not require the same rate.

Consumer backpressure

The renderer currently uses ReadableStream.desiredSize as an overload signal.

However, WebCodecs may accept a packet by synchronously calling VideoDecoder.decode() while retaining the actual decode work in its internal queue. Stream backpressure may therefore not represent the decoder and renderer workload.

Frame freshness

The renderer currently has no explicit packet freshness contract:

• no monotonic packet sequence;
• no sidecar receive/send timestamps;
• no maximum packet-age policy;
• no explicit recovery state after a sequence discontinuity.

Without these signals, the renderer cannot distinguish a current frame from an obsolete frame that was delayed in transport or renderer scheduling.

Keyframe recovery metadata

The renderer needs reliable keyframe metadata to recover after selectively dropping compressed frames.

ScrcpyMediaStreamPacket exposes keyframe, while the current wire protocol uses keyFrame. A typed mapping between these fields would make the recovery contract explicit and prevent metadata from being lost.

Proposed capability

1. Bound preview production

Apply an explicit frame-rate limit to the Studio preview, for example:

maxFps: 30

The existing preview size and bitrate can remain unchanged.

2. Add transport metadata

Attach lightweight metadata to each preview packet:

{
  sequence: number;
  receivedAt: number;
  sentAt: number;
  keyFrame?: boolean;
}

This metadata would support diagnostics and renderer-side admission decisions.

3. Add renderer admission control

Before submitting compressed data to WebCodecs, maintain a small state machine:

forwarding
  -- stale packet or sequence discontinuity --> waiting-for-keyframe

waiting-for-keyframe
  -- delta packet --------------------------> discard
  -- stale keyframe ------------------------> discard
  -- fresh keyframe ------------------------> forwarding

A packet-age threshold such as 500 ms could be used as the initial bounded-latency policy.

Configuration packets required for decoder initialization should continue to be forwarded.

4. Preserve keyframe information

Map the typed scrcpy field to the existing renderer wire field:

keyFrame: packet.type === 'data' ? packet.keyframe : undefined

This preserves wire compatibility while making keyframe recovery reliable.

5. Add preview diagnostics

Expose interval metrics for:

• sidecar read FPS and bytes;
• Socket.IO transport state;
• packet age and sequence discontinuities;
• stale-frame drops;
• time waiting for a keyframe;
• keyframe recoveries;
• decoded and rendered frame counts;
• render gaps and draw duration.

Expected outcome

With this capability:

• Preview latency remains bounded during temporary overload.
• The producer does not generate unnecessary preview work.
• Obsolete compressed frames are discarded before WebCodecs.
• Sequence loss results in a bounded wait for a fresh keyframe.
• The preview returns to the current device state after congestion clears.
• Image corruption is avoided when frames are selectively discarded.
• Existing scrcpy, Socket.IO, Web Streams, and WebCodecs architecture remains unchanged.

Suggested acceptance criteria

• Effective preview production does not exceed the configured frame-rate limit.
• Packet age does not grow continuously during dynamic content.
• Packets older than the configured freshness threshold are not submitted to WebCodecs.
• A sequence discontinuity causes delta frames to be discarded until a fresh keyframe arrives.
• Configuration packets retain decoder initialization behavior.
• Preview latency returns to below 500 ms after congestion clears.
• Memory use remains bounded during a 30-minute preview session.
• Existing callers without the additional transport metadata retain their current behavior.

Reproduce scenario

1. Open Midscene Studio.
2. Connect an Android physical device.
3. Wait for the scrcpy preview to become connected.
4. Open a screen with continuous scrolling, animation, or frequent UI changes.
5. Run an AI use case at the same time to add normal Studio renderer load.
6. Compare the physical device and Studio preview over several minutes.

Related work

• #2923 introduces frame-freshness admission for the Android screenshot and planning path. The proposed feature applies a similar bounded-freshness principle to the Studio live preview pipeline.
• #2911 discusses Recorder and live preview using different frame sources. This request focuses on flow control inside the live preview itself.

Reproduce link

None. A physical Android device and Midscene Studio are required.

What does the proposed API look like?

none

Start by tracing the Android Studio scrcpy preview path from the android-playground sidecar through Socket.IO, ReadableStream, WebCodecs VideoDecoder, and the preview canvas. Compare the proposed freshness and keyframe behavior with related issues #2923 and #2911; done means bounded latency, recovery after sequence loss, preserved decoder initialization, bounded memory, and the listed diagnostics.