MemberJunction / MemberJunction/MJ

[Computer Use] Ambiguous LLM-supplied selectors hard-fail on Playwright strict mode, cascading into false LoopDetected failures

Open
#3,318 0 comments 0 reactions 0 assignees View on GitHub
bug priority: high
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
2d 1h
Merged PRs (30d)
323

Description

## Summary

When the Computer Use controller (the LLM) supplies a `Selector` on an action, the browser adapters pass it straight to Playwright APIs that operate in **strict mode** — `page.click(selector)`, `locator.focus()`, `locator.scrollIntoViewIfNeeded()`. If the selector matches more than one element, Playwright throws instead of acting:

```
action Scroll failed: locator.scrollIntoViewIfNeeded: Error: strict mode violation:
locator('div:has-text("VISIBLE COLUMNS")') resolved to 14 elements
```

This is not an exotic case. `:has-text()` matches **every ancestor** that contains the text, so any `div:has-text("…")` the model writes is essentially guaranteed to match a whole chain of nodes. A bare `text=Foo` matches every node containing that text.

## Impact

From regression run `run-20260727T202606Z` (155 tests, 130 passing):

- **15 strict-mode violations** and **25 `page.click: Timeout` failures** across the run
- Concentrated in ~10 failing tests: T085–T089 (Credentials), T118 (File Browser), T141–T143 (Archiving / Realtime), T099 (Chat tabs)

The failure is worse than a single lost action. A failed action means the page state doesn't change, so the engine's loop detector counts a repeat state; three trips and the run terminates as `Failed / LoopDetected` — even though the agent was navigating correctly and only needed one of the matched nodes.

Actual selectors observed failing:

| Selector | Matched |
|---|---|
| `:has-text("Integrations")` | 29 |
| `.p-sidebar, .query-info, app-query-info, div:has-text("Query Information")` | 32 |
| `div:has-text('AssociationDemo')` | 17 |
| `.taxonomy-details-panel, div:has-text("GOVERNANCE")` | 15 |
| `div:has-text("VISIBLE COLUMNS")` | 14 |
| `.configuration-card, [class*="config"], div:has-text("Configuration")` | 14 |
| `div:has-text("MJ: AI Agent Configurations")` | 10 |
| `text=Integrations` | 4 |
| `.mat-drawer-inner-container, .slide-out-panel, .drawer-content, div[class*='slide']` | 4 |
| `text="Integrations"`, `text=Permissions`, `text=AssociationDemo` | 2–3 |

## Affected code

The selector branches are duplicated across both adapters:

- `packages/AI/ComputerUse/src/browser/PlaywrightBrowserAdapter.ts` — `Click` (~L733), `Type` (~L748), `Scroll` (~L810)
- `packages/AI/ComputerUse/src/browser/SharedContextBrowserAdapter.ts` — `Click` (~L284), `Type` (~L337), `Scroll` (~L364)

(`Wait` uses `page.waitForSelector`, which is **not** strict — it resolves the first match — so it is unaffected.)

## Expected behavior

An ambiguous selector should be **disambiguated, not fatal**. Playwright's own non-strict APIs resolve to the first match; for the ancestor-chain case that `:has-text()` produces, the *innermost* (smallest) match is the semantically correct target.

Proposal: resolve the selector to a single locator before acting —
- 0 or 1 matches → behave exactly as today (preserving auto-wait for elements that haven't rendered yet)
- more than 1 → prefer visible matches, then pick the smallest by bounding-box area (the innermost element of an ancestor chain), tie-breaking on document order

Because the multi-match case is currently a guaranteed hard failure, this cannot regress any presently-working action.

## Related

The prompt side is worth a follow-up too: the controller could be steered toward element-grounded actions (`ClickElement` / `TypeIntoElement`, which resolve by index against extracted elements and don't have this problem) rather than free-form CSS. But the engine should be robust regardless of what the model emits.

---
*Found by the MJ Explorer Computer Use regression suite, run `run-20260727T202606Z` (2026-07-27).*

Contributor guide

Open the contributing guide

Research direction

Start with the Click, Type, and Scroll branches in packages/AI/ComputerUse/src/browser/PlaywrightBrowserAdapter.ts and SharedContextBrowserAdapter.ts; compare their selector handling with the unaffected Wait path. Reproduce the ambiguous-selector failures from the MJ Explorer regression run, then verify zero, one, and multiple matches preserve auto-wait and select the intended visible target without strict-mode or LoopDetected failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
playwright, typescript
Domain
devtools, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.