MarketSquare / MarketSquare/robotframework-browser
Report the retry budget in assertion failure messages
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 655
- Forks
- 147
- Avg merge
- 5h 38m
- Merged PRs (30d)
- 58
Description
Is your feature request related to a problem? Please describe.
When an assertion keyword fails, the failure message says nothing about the fact
that the assertion was retried, or for how long. Get Element States id=not_here contains hidden
fails with:
Elements states '['detached']' (list) should contain '['hidden']' (list)
That message is identical whether the assertion failed instantly or was retried for a
full second before giving up. There is no hint that retrying happened, that it is
configurable, or that Set Retry Assertions For exists.
This is a discoverability problem more than a debugging one. Users who do not already
know about retry_assertions_for have no path to discovering it, because the one moment
they would care about it — an assertion that failed after quietly retrying — is exactly
the moment the library stays silent. Anecdotally, experienced Robot Framework users are
surprised the setting exists at all.
The information is already computed. Browser/assertion_engine.py logs it at DEBUG in the
finally block of with_assertion_polling:
Assertion polling statistics:
First element asserted in: 0.2534 seconds
Total tries: 4
Elapsed time in retries 0.7864 seconds
DEBUG is the wrong level for something a user needs in order to know a feature exists.
Describe the solution you'd like
Include the retry budget in the assertion failure message itself, at the two raise sites
in with_assertion_polling (Browser/assertion_engine.py).
Gate it on retries having actually happened (tries > 1). That is both the only case where
the information is useful and the thing that keeps the change small: an assertion that failed
on its first attempt, or one running with retry_assertions_for=0, keeps its message
byte-for-byte.
Describe alternatives you've considered
- Leave it at DEBUG. Status quo. Does not solve discoverability — nobody turns on DEBUG
before they know there is something to look for. - Document it harder. Does not reach the user at the moment they need it.
- Always append, ungated. Larger compatibility surface for no extra benefit, since a
non-retried assertion has no retry budget worth reporting.
Additional context
Compatibility. Appending to assertion messages is a breaking change for
Run Keyword And Expect Error patterns that match exactly. Measured in this repository's
own acceptance tests:
Run Keyword And Expect Error sites |
225 |
|---|---|
| pattern uses a glob / regexp | 141 |
| exact match | 84 |
The tries > 1 gate narrows the affected set considerably — the exact-match sites are
mostly non-retrying errors (Error: Crash, plugin errors) rather than assertion-keyword
failures. The residual should be measured before implementing. The same consideration
applies to users' suites, which is why this needs release notes.
Design question — where the information goes. The blast radius depends on placement, and
that is worth deciding deliberately rather than defaulting to "append to the end":
- appended as a trailing sentence — simplest, but touches the message that exact matches see;
- a separate
WARN/INFOlog line beside the failure — zero message compatibility cost,
but easier for a user to miss; - attached only when the assertion exhausted its budget (as opposed to any retry at all).
Secondary benefit. retry_assertions_for is currently the one scope setting whose effect
cannot be verified except with a stopwatch. atest/test/08_Scope_Tests measures wall-clock
duration to infer it, and that is the source of roughly 20 CI failures across the last ~1200
runs. Every other scope setting can be verified from a message — Playwright names the timeout
it used (Timeout 1500ms exceeded), and strict mode raises strict mode violation. Naming
the retry budget in the failure would let those tests assert on a message too, and drop the
last timing-based assertion in that suite.
Related: Browser/assertion_engine.py, atest/test/08_Scope_Tests/scope_keywords.resource.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in Browser/assertion_engine.py, reading with_assertion_polling and its two raise sites alongside the existing DEBUG statistics. Review atest/test/08_Scope_Tests/scope_keywords.resource and the repository's acceptance tests to assess message compatibility and placement. Done means retried assertion failures expose the retry budget while first-attempt failures retain their existing messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- playwright, python
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100