Android device panel never leaves "Connecting video…": the keyframe request after decoder configure restarts the encoder, which tears the decoder back down

What happened

The Device panel opens, the device is listed and is fully interactive — taps and
swipes land on the emulator, device_screenshot returns real frames, and
agent-device drives the device fine — but the video area sits on
"Connecting video…" forever and never paints a single frame. Restarting the
panel, the hub, the emulator and the server changes nothing.

Diagnosis

Android video never reaches status: "streaming" because the client and
serve-emu deadlock on the keyframe handshake. Neither side is broken on its own;
the combination never converges.

The client's Android path (apps/web/src/components/device/deviceStream.ts):

1. ws.onmessage — a video-session text message means "the encoder restarted
   at a new size (device rotation)", so it calls closeDecoder():

   if (isVideoSessionMessage(event.data)) closeDecoder();
   

2. The first keyframe carries the SPS. Since no decoder is configured yet, the
   client configures from that SPS, then drops that keyframe and asks for a
   fresh one (deviceStream.ts:566-574):

   if (scanned?.sps && (!videoDecoder || videoDecoder.state !== "configured")) {
     if (configuring) return;
     configuring = true;
     void configureDecoder({ codec: avcCodecString(scanned.sps) }).then((configured) => {
       configuring = false;
       awaitingKeyframe = true;
       if (configured) requestKeyframe();
     });
     return;
   }
   

3. requestKeyframe() sends {"type":"reset-video","ack":false}.

serve-emu's side (expo-device-hub 0.9.0,
vendor/serve-emu/dist/middleware.js, the Node middleware build that t3 mounts
under /vendor/serve-emu):

4. reset-video → requestVideoReset("client requested keyframe") →
   performVideoReset → session.controls.enqueueVideoReset(), which writes
   scrcpy's RESET_VIDEO control packet. scrcpy answers by restarting its
   encoder, not by emitting an IDR into the existing stream.

5. The restarted encoder produces a new session frame. The capture reader
   broadcasts video-session to every video client — gated only on
   f.width > 0 && f.height > 0, so an unchanged 576x1280 restart is announced
   exactly like a rotation (middleware.js:603-624):

   if (f.type === "session") {
       if (f.width > 0 && f.height > 0) {
           ...
           for (const c of clients) {
               if (!c.video) continue;
               c.awaitingKeyFrame = true;
               sendJson(c.socket, { type: "video-session", size: { width: f.width, height: f.height } });
           }
       }
       continue;
   }
   

6. That video-session hits step 1 → closeDecoder() → the next keyframe again
   finds no configured decoder → step 2 configures and requests a keyframe again
   → step 4. Forever.

The period is VIDEO_RESET_COOLDOWN_MS = 1500 (middleware.js:43), which is
exactly the ~1.5 s cycle observed on the wire. paint() is never reached, so
firstFrame stays false and setStatus("streaming") is never called — the
panel's label is literally correct: it is still connecting.

Note the client's own comment at the video-session branch assumes that message
only ever means rotation. serve-emu sends it for any encoder restart,
including the one the client itself just asked for.

Verified on the wire, on this machine

Replaying connectAndroid()'s exact state machine (SEMU parse, scanAccessUnit,
video-session → close, configure → reset-video) against the live hub, with
the real VideoDecoder stubbed out:

000008 OPEN
000066 <- TEXT {"type":"video-session","size":{"width":576,"height":1280}}   [closeDecoder()]
000118 <- KEY len=26377 sps -> configureDecoder(), then reset-video
000118 -> reset-video (#1)
... deltas ...
001145 <- TEXT {"type":"video-session","size":{"width":576,"height":1280}}   [closeDecoder()]
001212 <- KEY len=26377 sps -> configureDecoder(), then reset-video
001213 -> reset-video (#2)
002658 <- TEXT {"type":"video-session","size":{"width":576,"height":1280}}   [closeDecoder()]
002696 <- KEY len=26377 sps -> configureDecoder(), then reset-video
002696 -> reset-video (#3)
(continues; 8 cycles in 20 s, 0 frames ever handed to the decoder)

Control run, identical except that the post-configure reset-video is not sent —
the SPS-carrying keyframe is decoded instead of dropped:

000069 <- video-session (#1) closeDecoder()
000120 <- KEY len=26371 sps=6742c029 codec=avc1.42c029 -> configure (NO reset-video)
000120    DECODE this same keyframe -> paint -> status=streaming
000205 <- frame len=352 key=false decoded, painted=2
000306 <- frame len=293 key=false decoded, painted=3
...
SUMMARY 20s (no reset-video): frames=11 painted=11 dropped=0 video-session msgs=1

So the stream itself is healthy: baseline H.264 avc1.42C029, SEMU v1 framing,
valid Annex-B, one decodable keyframe up front. The only thing standing between
the panel and a picture is the keyframe request it sends after configuring.

The hub's own counters show the real desktop client doing the same thing, not
just the replay — videoResetRequests tracks 1:1 with configPackets, and the
reason string is the one only requestKeyframe() produces:

"configPackets": 424, "videoResetRequests": 423,
"lastVideoResetReason": "client requested keyframe",
"streamMode": "scrcpy", "inputSource": "scrcpy", "codec": "h264",
"size": {"width":576,"height":1280}, "sessionGeneration": 1,
"frames": 5261, "droppedFrames": 9, "backpressureEvents": 0

sessionGeneration: 1 confirms the capture session is never replaced — this is
scrcpy's encoder restarting inside one session, which is precisely the case the
client misreads as a rotation.

Where a fix would go

Any one of these breaks the loop; the first is the smallest:

• Decode the SPS-carrying keyframe instead of dropping it, and only request a
  keyframe when the first frame after configure is not a keyframe
  (deviceStream.ts:566-574). serve-emu already sends config+keyframe as one
  message, so the fresh keyframe the client asks for is redundant.
• Ignore a video-session whose size equals the current canvas size, rather
  than tearing down the decoder on every one.
• Upstream in serve-emu: only broadcast video-session when the size actually
  changed, so a client-requested reset is not announced as a new session.

Steps to reproduce

1. Linux or WSL2 host with an Android emulator booted and adb working.
2. Run the t3 server, open the Device panel for the emulator from the desktop
   app (any surface — the panel is the same bundle).
3. Make sure the hub's Android capture source is scrcpy
   (GET /vendor/serve-emu/api/stream-mode?device=<serial> → "mode":"scrcpy").
4. Watch the panel: it stays on "Connecting video…" indefinitely. Input works,
   screenshots work.
5. Confirm the loop: GET /vendor/serve-emu/health?device=<serial> and poll —
   videoResetRequests climbs by 1 roughly every 1.5 s while the panel is open,
   lastVideoResetReason is "client requested keyframe", and
   configPackets == videoResetRequests + 1.

Deterministic on this machine: every panel open reproduces it, and both replay
scripts above reproduce it against the live hub without the panel involved.

Version

0.0.43-nightly.20260917.1837 (server). npx t3 triage reported installed
version 0.0.42; the running server binary is the nightly above. Code references
checked against main @ ccf220be, which matches the shipped client bundle
byte for byte in the relevant functions.

Environment

WSL2 (Linux 6.6.87.2-microsoft-standard-WSL2, x64), Node v26.8.2 for the t3
CLI, hub spawned under Node v24.18.0. Desktop app T3Code(Nightly)/0.0.43,
Electron/44.1.0, running on the Windows host against the WSL server at
127.0.0.1:3775. expo-device-hub 0.9.0 (pinned in
apps/server/src/device/DeviceToolchain.ts:28), Android emulator 37.1.11,
AVD Pixel_10, scrcpy server 4.0.

Evidence

# serve-emu health, while the panel was open (redacted, trimmed)
{"ok":true,"status":"streaming","captureRestarting":false,
 "streamMode":"scrcpy","inputSource":"scrcpy","codec":"h264",
 "size":{"width":576,"height":1280},"clients":0,"videoClients":0,
 "frames":5261,"configPackets":424,"droppedFrames":9,"backpressureEvents":0,
 "videoResetRequests":423,"lastVideoResetAt":"2026-09-17T09:12:49.077Z",
 "lastVideoResetReason":"client requested keyframe",
 "frameStats":{"windowFrames":240,"intervalMs":{"p50":101,"p95":496,"max":59161},
   "avgKeyFrameBytes":26331,"avgDeltaFrameBytes":7035,"keyFramesInWindow":13},
 "sessionGeneration":1}

# first binary message on ws /vendor/serve-emu/ws?device=<serial>&frame-meta=1
BIN#0 len=26359 first32=53454d550101000000000001395f8536000000016742c0298d680900a1a42020
   magic=SEMU version=1 flags=0x1(key) reserved=0 pts=5257528630
   payload[0..]=00000001 6742c029 ...   # Annex-B start code + SPS (NAL type 7)
   -> avcCodecString(sps) = avc1.42c029, baseline, decodable

# client state machine replayed against the live hub (VideoDecoder stubbed)
000118 <- KEY len=26377 sps -> configureDecoder(), then reset-video
000118 -> reset-video (#1)
001145 <- TEXT {"type":"video-session","size":{"width":576,"height":1280}}   [closeDecoder()]
001212 <- KEY len=26377 sps -> configureDecoder(), then reset-video
001213 -> reset-video (#2)
002658 <- TEXT {"type":"video-session","size":{"width":576,"height":1280}}   [closeDecoder()]
002696 <- KEY len=26377 sps -> configureDecoder(), then reset-video
002696 -> reset-video (#3)
SUMMARY after 20s: video-session msgs=8  reset-video sent=8  frames painted=0

# same replay, post-configure reset-video removed
000120 <- KEY len=26371 sps=6742c029 codec=avc1.42c029 -> configure (NO reset-video)
000120    DECODE this same keyframe -> paint -> status=streaming
SUMMARY 20s (no reset-video): frames=11 painted=11 dropped=0 video-session msgs=1

# hub spawn (t3 server -> expo-device-hub)
node .../expo-device-hub/dist/server/cli.mjs --port 38457 --host 127.0.0.1 \
  --hide-sidebar --hide-boot-device

# relevant constants
middleware.js:40  FRAME_META_VERSION = 1
middleware.js:41  FRAME_META_HEADER_BYTES = 16
middleware.js:43  VIDEO_RESET_COOLDOWN_MS = 1500

Related issues

#12219 (closed) — same machine, the layer underneath this one: the hub never
started at all on WSL because resolveNodeExecutable handed back the t3 binary
as nodePath. With that fixed the hub starts, streams, and accepts input; this
issue is what is left, and it is a separate defect in the client's keyframe
handshake rather than a spawn problem. No open issue found matching
"Connecting video", reset-video or video-session.

Fix applied or workaround

No workaround found for this issue — the loop is unconditional on the scrcpy
capture source, and the panel cannot be coaxed into streaming from the outside.

One earlier local change is still in place on this machine and should be
accounted for when reading the evidence above: the installed hub's
dist/server/cli.mjs was replaced with a two-line shim that appends
--stream-source scrcpy before importing the real CLI (kept as
cli.real.mjs), because t3 spawns the hub with only
--port/--host/--hide-sidebar/--hide-boot-device and the hub's default
grpc-screenshot source produced no frames on this machine. Reverting is
mv cli.real.mjs cli.mjs inside the installed hub package. So the streaming
path measured here is scrcpy; whether the same loop occurs under
grpc-screenshot was not tested, since that source never produced a frame here
in the first place. That default-source behaviour is a separate question and is
deliberately not mixed into this report.

Filed by

claude (opus-5) via t3 triage

Start in apps/web/src/components/device/deviceStream.ts:566-574 and trace configureDecoder, requestKeyframe, and the video-session handler while reproducing the panel loop. Compare this with vendor/serve-emu/dist/middleware.js:603-624 and the reset-video path. Done means the initial SPS-carrying keyframe is decoded and painted, the panel reaches streaming, and repeated video resets stop.