MemberJunction / MemberJunction/MJ
[Computer Use] Ambiguous LLM-supplied selectors hard-fail on Playwright strict mode, cascading into false LoopDetected failures
- Dominant language
- TSQL
- Stars
- 29
- Forks
- 6
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 323
Description
## Summary
When the Computer Use controller (the LLM) supplies a `Selector` on an action, the browser adapters pass it straight to Playwright APIs that operate in **strict mode** — `page.click(selector)`, `locator.focus()`, `locator.scrollIntoViewIfNeeded()`. If the selector matches more than one element, Playwright throws instead of acting:
```
action Scroll failed: locator.scrollIntoViewIfNeeded: Error: strict mode violation:
locator('div:has-text("VISIBLE COLUMNS")') resolved to 14 elements
```
This is not an exotic case. `:has-text()` matches **every ancestor** that contains the text, so any `div:has-text("…")` the model writes is essentially guaranteed to match a whole chain of nodes. A bare `text=Foo` matches every node containing that text.
## Impact
From regression run `run-20260727T202606Z` (155 tests, 130 passing):
- **15 strict-mode violations** and **25 `page.click: Timeout` failures** across the run
- Concentrated in ~10 failing tests: T085–T089 (Credentials), T118 (File Browser), T141–T143 (Archiving / Realtime), T099 (Chat tabs)
The failure is worse than a single lost action. A failed action means the page state doesn't change, so the engine's loop detector counts a repeat state; three trips and the run terminates as `Failed / LoopDetected` — even though the agent was navigating correctly and only needed one of the matched nodes.
Actual selectors observed failing:
| Selector | Matched |
|---|---|
| `:has-text("Integrations")` | 29 |
| `.p-sidebar, .query-info, app-query-info, div:has-text("Query Information")` | 32 |
| `div:has-text('AssociationDemo')` | 17 |
| `.taxonomy-details-panel, div:has-text("GOVERNANCE")` | 15 |
| `div:has-text("VISIBLE COLUMNS")` | 14 |
| `.configuration-card, [class*="config"], div:has-text("Configuration")` | 14 |
| `div:has-text("MJ: AI Agent Configurations")` | 10 |
| `text=Integrations` | 4 |
| `.mat-drawer-inner-container, .slide-out-panel, .drawer-content, div[class*='slide']` | 4 |
| `text="Integrations"`, `text=Permissions`, `text=AssociationDemo` | 2–3 |
## Affected code
The selector branches are duplicated across both adapters:
- `packages/AI/ComputerUse/src/browser/PlaywrightBrowserAdapter.ts` — `Click` (~L733), `Type` (~L748), `Scroll` (~L810)
- `packages/AI/ComputerUse/src/browser/SharedContextBrowserAdapter.ts` — `Click` (~L284), `Type` (~L337), `Scroll` (~L364)
(`Wait` uses `page.waitForSelector`, which is **not** strict — it resolves the first match — so it is unaffected.)
## Expected behavior
An ambiguous selector should be **disambiguated, not fatal**. Playwright's own non-strict APIs resolve to the first match; for the ancestor-chain case that `:has-text()` produces, the *innermost* (smallest) match is the semantically correct target.
Proposal: resolve the selector to a single locator before acting —
- 0 or 1 matches → behave exactly as today (preserving auto-wait for elements that haven't rendered yet)
- more than 1 → prefer visible matches, then pick the smallest by bounding-box area (the innermost element of an ancestor chain), tie-breaking on document order
Because the multi-match case is currently a guaranteed hard failure, this cannot regress any presently-working action.
## Related
The prompt side is worth a follow-up too: the controller could be steered toward element-grounded actions (`ClickElement` / `TypeIntoElement`, which resolve by index against extracted elements and don't have this problem) rather than free-form CSS. But the engine should be robust regardless of what the model emits.
---
*Found by the MJ Explorer Computer Use regression suite, run `run-20260727T202606Z` (2026-07-27).*
Contributor guide
Research direction
Start with the Click, Type, and Scroll branches in packages/AI/ComputerUse/src/browser/PlaywrightBrowserAdapter.ts and SharedContextBrowserAdapter.ts; compare their selector handling with the unaffected Wait path. Reproduce the ambiguous-selector failures from the MJ Explorer regression run, then verify zero, one, and multiple matches preserve auto-wait and select the intended visible target without strict-mode or LoopDetected failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- playwright, typescript
- Domain
- devtools, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100