[browser-use] IAB guest input dispatch & screenshot silently dropped when pane loses foreground; CDP ops unaffected

TL;DR (中文)

内置浏览器(IAB)的 agent 输入事件(cua.click / dom_cua.click / locator.click / keypress)与 screenshot 在面板失焦/guest 节流时被静默丢弃(实测成功率 ~10-20%),而同期 CDP 类操作(goto / fill / domSnapshot)100% 可靠。建议输入事件改走 CDP Input domain,失败时显式报错而非静默丢弃。

Environment

• ZCode Desktop, Windows 11
• browser-use plugin 0.4.0 (zcode-plugins-official), backend: Desktop IAB
• Target: local React (vite) dev app; agent driving a full E2E flow

Symptoms

• cua.click / dom_cua.click: command returns, but page state unchanged (React onClick not fired) — ~80-90% of attempts
• playwright.locator.click(): always times out even with force: true (locator count() resolves fine in the same cell)
• tab.screenshot(): never succeeded in the whole session: browser screenshot activity capture failed for guest / browser screenshot surface preparation timed out after 3000ms
• Intermittent: browser guest not attached (webview not ready), browser guest changed before command dispatch
• locator.press("Enter") on buttons: no effect; locator.fill() always works

What keeps working (same tab, same moment)

• tab.goto() / tab.reload() / playwright.fill() / playwright.domSnapshot() / dom_cua.get_visible_dom() — 100% reliable

Repro

1. browser.tabs.new() → load a page with a button that toggles observable UI state (e.g. opens a modal)
2. dom_cua.get_visible_dom() → take element rect → cua.click at center
3. Verify via domSnapshot() → ~8-9 of 10 attempts show no effect
4. Success rate rises sharply right after browser.tabs.get(id) re-activation, or when the user manually focuses/interacts with the IAB pane; drops to ~0 after the pane loses foreground (user switches away/minimizes)

Expected

• Input events (mouse/keyboard) dispatched via CDP (Input.dispatchMouseEvent / Input.dispatchKeyEvent) without depending on the webview rendering/activity pipeline
• Dispatch failures surface as explicit errors instead of silent drops
• Screenshot activity capture should work in agent sessions

Workaround that made E2E possible

Before every input: re-activate via tabs.get(id), single-shot click, immediately verify effect via domSnapshot()/backend logs, retry on failure. Route everything possible through goto/fill (CDP path).

Start by tracing the browser.tabs.get reactivation path and the cua.click, dom_cua.click, locator.press, and tab.screenshot entry points described in the report. Compare these with the reliable goto, fill, and domSnapshot paths and the reported CDP Input operations. Done means input is not silently dropped when the pane loses focus, failures are explicit, and screenshot capture works in agent sessions.