microsoft / microsoft/playwright
[CLI]: skill doesn't warn that a generated locator is only unique against the current page state
- Dominant language
- TypeScript
- Stars
- 96.3k
- Forks
- 6.5k
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 180
Description
### Version
Bundled skill from `@playwright/cli@0.1.20`. Verified against `main`:
`packages/playwright-core/src/tools/skills/playwright-cli/references/test-generation.md`
is byte-identical to the copy that ships in the package, so this applies to the
current source.
### Problem
`references/test-generation.md` §0 tells the generating agent to use
`generate-locator` to produce the locator for an assertion, and `recording-stop`
to turn a drive into test code. Neither says that both emit **the shortest
locator that is unambiguous on the page as it stands at that moment**.
An agent that generates a locator while the page happens to be filtered,
paginated, or in an empty state gets a locator that is silently wrong the moment
a sibling reappears — and nothing in the skill tells it that this is a thing that
can happen.
### Reproduction
Against `https://demo.playwright.dev/todomvc`, one session, one element:
```bash
playwright-cli open https://demo.playwright.dev/todomvc
playwright-cli snapshot
playwright-cli fill e8 "buy milk"
playwright-cli press Enter
playwright-cli fill e8 "write regression test"
playwright-cli press Enter
playwright-cli snapshot # e21 = "Toggle Todo" checkbox of the "buy milk" row
playwright-cli check e21
# Ran Playwright code:
# await page.getByRole('listitem').filter({ hasText: 'buy milk' }).getByLabel('Toggle Todo').check();
playwright-cli click e32 # the "Completed" filter — only one row is now visible
playwright-cli --raw generate-locator e21
# getByRole('checkbox', { name: 'Toggle Todo' }) <-- ambiguous
playwright-cli click e28 # back to "All" — both rows visible again
playwright-cli snapshot
playwright-cli --raw generate-locator e21
# getByRole('listitem').filter({ hasText: 'buy milk' }).getByLabel('Toggle Todo')
```
Same element, same session, two different locators — and the one produced from
the filtered view no longer identifies it once the sibling is back.
The consequence shows up at replay. Running the recorded flow a second time
without resetting storage leaves a second `buy milk` row, and the **recorded**
line — not an assertion — fails with:
```
strict mode violation: getByRole('listitem').filter({ hasText: 'buy milk' })
.getByLabel('Toggle Todo') resolved to 2 elements
```
Strict mode is doing the right thing here; the point is that the skill never
tells the agent this failure mode exists, so the agent has no reason to record
with the full working set on screen or to reset state before replay.
### Why the current text doesn't cover it
The neighbouring concern *is* handled structurally — §1.2 seed tests, and §2.3
"restarting the seed between each so every test starts from a clean page" — but
it is framed as scenario independence, never as "the locator you just generated
encodes the page state it saw". §0 opens by saying the sections can be used
independently, and an agent reading §0 alone gets no signal at all.
Separately: across the whole `playwright-cli` skill tree on `main` there are zero
occurrences of `strict mode`, `ambiguous`, or `resolved to N`. A generating agent
has never been told what that failure will look like, so when it hits one it has
no reason to connect it back to how the locator was produced. The likely
"fix" it reaches for is `.first()`.
### Suggested fix
A short paragraph in §0, after **Use semantic locators**. Roughly:
> **A generated locator is unique only against the state it was generated in.**
> `generate-locator` and `recording-stop` emit the shortest locator that is
> unambiguous on the page as it stands at that moment. If the page is filtered,
> paginated or empty when you generate, the locator can stop identifying the
> element once a sibling appears, and the action will fail with
> `strict mode violation: ... resolved to N elements`. Generate with the full
> working set on screen, and start each replay from the seed's clean state.
> Do not resolve a strict-mode violation with `.first()` — regenerate the
> locator with the siblings visible.
Happy to send that as a PR if you would like it from the community; filing the
issue first per CONTRIBUTING.
Contributor guide
Research direction
Edit packages/playwright-core/src/tools/skills/playwright-cli/references/test-generation.md, focusing on §0 after “Use semantic locators.” Read the existing guidance alongside the reproduction and suggested paragraph. Done means §0 explains state-dependent locator uniqueness, strict-mode failures, full working-set generation, clean replay state, and avoiding `.first()`.
Written by the indexing model from the issue text.
Assessment
- Domain
- cli, documentation, testing-qa
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 91/100