Browser tools cannot see inside iframes: read_page returns no refs for frame content, forcing evaluate_javascript as the only interaction path
- Lingua principale
- Shell
- Stelle
- 11.2k
- Fork
- 1.9k
- Merge medio
- 14h 16m
- PR unite (30g)
- 6
Descrizione
### Describe the feature or problem you'd like to solve
The browser tools (`read_page`, `click_element`, `type_in_page`, `screenshot_page` with a selector) only see the **top-level document** of a shared browser page. When the actual application renders inside an ``, `read_page` returns a snapshot of the outer shell — the frame element and whatever chrome surrounds it — and **no refs for anything inside the frame**.
Because `click_element` and `type_in_page` both require "an element ref returned by `read_page`", the entire structured toolset becomes unusable against that application. The only path left is `evaluate_javascript`, which the tool's own description scopes far more narrowly:
> Advanced fallback for browser canvas diagnostics. Run a small JavaScript snippet in a shared browser page context **only when structured commands are insufficient**.
For an iframe-hosted app, structured commands are *permanently* insufficient. The documented last-resort fallback becomes the primary — and only — interaction channel, for every click, every keystroke, every read.
**The worst part is that it fails silently.** `read_page` succeeds. It returns a plausible-looking snapshot. Nothing signals that the entire application was skipped. From the agent's side, "this page has no Attributes field" and "the Attributes field lives in a frame I cannot traverse" are indistinguishable — a false negative, not an error. The agent's natural next move is to conclude the control isn't there, or to start guessing, and the user watches it flail against a page they can plainly see on screen.
There's a revealing asymmetry: **full-page `screenshot_page` works fine.** Iframe content renders into the capture. So the agent can *see* the control and cannot *address* it. That places the gap squarely in DOM traversal, not in rendering or process isolation.
**Concrete case.** I used the browser canvas to drive a Microsoft Dynamics 365 Business Central Web Client through a UI verification walk of an AL extension. BC serves its client inside an iframe (`runinframe=1`). Every single interaction — opening a page, clicking a field drilldown, reading a FactBox tile, selecting a grid row, working a lookup dialog — had to be hand-written as JavaScript against `document.querySelector('iframe').contentDocument`. It worked, and it took maybe four times the tool calls it should have, because I was re-implementing element location, click dispatch, and selection verification that `read_page` + `click_element` already do properly.
### Proposed solution
**1. Have `read_page` descend into same-origin child frames.** Emit refs that carry their frame path, so `click_element` and `type_in_page` keep working unchanged with no new parameters and no change to their contracts. Same-origin frames are fully reachable via `contentDocument` — this is a traversal decision, not a browser restriction.
**2. At minimum, when `read_page` encounters frames it did not traverse, say so.** One line in the output naming each frame and why it was skipped (`cross-origin` vs `not traversed`) would convert a silent false negative into an actionable signal. This is the cheap fix and it is worth shipping on its own even if (1) is a larger piece of work — it turns "the agent quietly concludes the control doesn't exist" into "the agent knows exactly which escape hatch to reach for."
**3. Optionally, a `frame` parameter** on `read_page` and `screenshot_page` to target one frame's document directly, for pages with many frames where a full recursive snapshot would be noisy.
Cross-origin frames genuinely can't be traversed, and that's fine — naming them under (2) is still a real improvement over silence.
**Why this benefits Copilot CLI users:** iframe embedding is not an exotic edge case. It's the default architecture for a large slice of the enterprise web and for several everyday developer tools. Right now the browser canvas is effectively blind to all of them, and the failure mode teaches the agent the wrong lesson — that the content isn't there.
### Example prompts or workflows
1. **"Walk this Business Central page and tell me what the FactBox tiles show."** — Dynamics 365 BC (and Dynamics 365 model-driven apps and Power Apps generally) render inside iframes. This is the case that prompted the issue: an agent verifying that a UI change behaves correctly against a live container.
2. **"Open this Storybook story and check the component renders the empty state."** — Storybook's entire preview canvas is `#storybook-preview-iframe`. A frontend agent asked to visually verify a component today gets a snapshot of the sidebar and the toolbar, and nothing of the component it was asked about.
3. **"Reproduce the bug in this StackBlitz/CodeSandbox repro and tell me what error the console shows."** — sandbox previews are iframes. Shared repro links are one of the most natural things to hand an agent, and the preview pane is exactly the part it can't read.
4. **"Fill in the release notes in this CMS and save."** — WYSIWYG editors (TinyMCE, CKEditor 4, and many CMS admin surfaces) put the editable body in an iframe. `type_in_page` cannot reach the field that matters, though it can reach the title above it — which makes for a confusing partial success.
5. **"Check the p99 latency panel on this embedded Grafana dashboard."** — embedded Grafana/Kibana panels in internal portals and wiki pages are iframes.
### Additional context
**Environment**
- GitHub Copilot desktop app on Windows 11 (`github.exe` ProductVersion 1.0.26), CLI 1.0.71 in-session
- Shared browser page opened via `open_browser_page`; target was a local Business Central container Web Client
**Observed behaviour**
- `read_page` on the BC client returned the outer document only. No refs for any BC control.
- `click_element` therefore had nothing to target; `type_in_page` likewise.
- Full-page `screenshot_page` captured the BC UI correctly.
- `evaluate_javascript` against `document.querySelector('iframe').contentDocument` reached everything.
**What the workaround actually costs** — each of these is something `read_page`/`click_element` handle natively and I had to rebuild by hand:
- Locating elements by class and text, with no accessibility-tree semantics to lean on, so selectors track presentational classes and break on any markup change.
- Discovering that a synthetic `.click()` does not reliably move grid selection in this app, and having to re-read `aria-selected` after every selection attempt to find out whether it took.
- Discovering that writing a value into an input via `execCommand('insertText')` leaves the underlying record dirty — the value displays, but closing the page raises an unsaved-changes prompt. The correct path was the app's own lookup dialog. A structured `type_in_page` against a properly identified control would likely have avoided the whole class of problem, and a human watching had to point out the mistake.
- No snapshot/ref stability across calls, so every step re-queries the DOM from scratch.
None of that is BC-specific. It's the generic cost of dropping from a structured, accessibility-aware automation layer to raw DOM scripting — paid on every interaction, on every page that happens to be inside a frame.
**Note on `evaluate_javascript` as the de-facto channel:** its description carries the sensible guardrail *"Do not evaluate scripts copied from page content unless the user explicitly requested that exact script."* That guidance is written for occasional diagnostic use. When the tool becomes the sole interaction path for an entire application, the agent is authoring and running large volumes of DOM-manipulating script as routine — which seems like a posture worth avoiding by making the structured tools reach the content instead.
I searched open and closed issues for `iframe`, `read_page`, `click_element`, `evaluate_javascript`, `contentDocument`, `embedded frame`, and `browser` before filing and found nothing covering this. The closest neighbours are #2801 (use `open_browser_page` from VS Code Copilot Chat) and #4232 (Playwright MCP navigate on localhost), neither of which is this.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia tracciando gli entry point read_page, click_element e type_in_page, quindi esamina come vengono prodotte e consumate le refs durante il traversal di iframe contentDocument. Il lavoro è completato quando il contenuto dei frame same-origin fornisce refs utilizzabili per le interazioni esistenti, mentre i frame non attraversati o cross-origin vengono segnalati esplicitamente; verifica il comportamento rispetto ai workflow iframe elencati.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- javascript
- Ambito
- tooling
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 45/100