microsoft / microsoft/playwright

[Bug]: [MCP] Accessibility tree snapshot includes off-screen/non-viewport elements causing AI agents to generate invalid test cases on unreachable UI elements

Open
#39,955 20 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

P3-collecting-feedback
Dominant language
TypeScript
Stars
96.3k
Forks
6.5k
Avg merge
1d 6h
Merged PRs (30d)
180

Description

### Version

1.49.0

### Steps to reproduce

1. Set up Playwright MCP and connect an AI agent to it
2. Navigate to https://wearecollins.com/story/TribecaFilm using browser_navigate
3. Call browser_snapshot to get the accessibility tree
4. Observe the snapshot output carefully
5. Notice footer elements like "Case Studies", "Programs", "LinkedIn",
"Work with us", "Team", "Careers" appear in the snapshot
6. These elements are NOT visible in the current viewport — they are
hundreds of pixels below the fold
7. An AI agent reads this snapshot and generates test cases for these
footer elements treating them as directly accessible

### Expected behavior

The accessibility tree snapshot should either:
1. Only return elements that are currently visible in the viewport, OR
2. Include a property on each element (e.g. "inViewport: true/false")
so AI agents can filter elements that are actually reachable
without scrolling

Elements that are off-screen, inside closed modals, or not reachable
without additional user actions should NOT be treated as directly
interactive elements.

### Actual behavior

The MCP accessibility tree snapshot returns ALL elements in the DOM
regardless of whether they are in the current viewport or not.

Example — on page load of https://wearecollins.com/story/TribecaFilm,
the snapshot includes:

- "Case Studies" link ← in footer, not in viewport
- "Programs" link ← in footer, not in viewport
- "Arts & Culture" link ← in footer, not in viewport
- "Work with us" button ← in footer, not in viewport
- "Team" button ← in footer, not in viewport
- "LinkedIn" link ← in footer, not in viewport
- Email input ← in footer, not in viewport

All of these pass through as valid interactive elements in the
accessibility tree even though a real user cannot interact with them
without first scrolling to the bottom of the page.

This causes AI agents to generate invalid test cases for unreachable
elements.

### Additional context

This issue directly impacts AI agent workflows built on Playwright MCP
for automated test case generation.

The core problem:
- Accessibility tree is designed to expose ALL elements for screen readers
- Screen readers can navigate non-linearly (jump to any element)
- AI agents need ONLY what a real user can see and click RIGHT NOW
- These two purposes conflict — the accessibility tree is not designed
for viewport-aware element discovery

Workaround we attempted:
Using page.evaluate() with getBoundingClientRect() to manually filter
viewport-visible elements:

const vh = window.innerHeight;
const vw = window.innerWidth;
Array.from(document.querySelectorAll('*')).filter(el => {
const r = el.getBoundingClientRect();
return r.top >= 0 && r.bottom <= vh && r.left >= 0 && r.right <= vw;
});

This is unreliable and requires manual maintenance per page.

Feature Request:
Add a viewport-only mode to browser_snapshot, for example:
browser_snapshot({ viewportOnly: true })

This would make Playwright MCP significantly more useful for
AI agent test generation use cases.

### Environment

```shell
System:
OS: Windows 11 10.0.26200
CPU: (12) x64 12th Gen Intel(R) Core(TM) i5-1235U
Memory: 6.13 GB / 23.69 GB
Binaries:
Node: 20.19.6 - C:\nvm4w\nodejs\node.EXE
npm: 10.8.2 - C:\nvm4w\nodejs\npm.CMD
IDEs:
VSCode: 1.110.0 - C:\Users\ZayedBinAliBabsail\AppData\Local\Programs\Microsoft VS Code\bin\code.CMD
Cursor: 1.7.54 - C:\Program Files\cursor\resources\app\bin\cursor.CMD
npmPackages:
@playwright/mcp: ^0.0.51 => 0.0.51
playwright: ^1.49.0 => 1.57.0
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with Playwright MCP's browser_snapshot on the linked TribecaFilm page and compare the returned footer elements with the current viewport. Read the browser_snapshot implementation and its accessibility-tree handling. Done means a clearly defined viewport-aware mode or element metadata, with coverage for off-screen and currently reachable elements.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.