GoogleChrome / GoogleChrome/lighthouse-ci
Retry failed assertions
- Dominant language
- JavaScript
- Stars
- 7.1k
- Forks
- 717
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
Due to [variability](https://developers.google.com/web/tools/lighthouse/variability?hl=en), Lighthouse CI may sometimes report a lower performance score that causes an assertion to fail, whereas for most runs, the assertion passes. Since the Lighthouse test takes rather long, I've configured `numberOfRuns: 1` for my CI pipeline. However, if, and ideally only if, a performance assertion fails, it would be helpful if the test could be re-run for a given URL.
Increasing `numberOfRuns` isn't quite the same thing, as it will run multiple times also when the first test passes.
**Describe the solution you'd like**
An option to retry the assertions for a given page if the the first run fails.
**Describe alternatives you've considered**
Wrapping the entire LHCI run to re-run if it fails (this needlessly tests URLs again that have already passed).
Contributor guide
Research direction
Start by tracing how Lighthouse CI runs a page URL, applies performance assertions, and uses numberOfRuns. Determine where a failed assertion could trigger a page-only retry without rerunning pages that already passed. Done means the retry option is defined, failed assertions retry as intended, and successful assertions do not cause extra runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- performance, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100