GoogleChrome / GoogleChrome/lighthouse-ci

Retry failed assertions

Open
#950 4 comments 2 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
7.1k
Forks
717
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**

Due to [variability](https://developers.google.com/web/tools/lighthouse/variability?hl=en), Lighthouse CI may sometimes report a lower performance score that causes an assertion to fail, whereas for most runs, the assertion passes. Since the Lighthouse test takes rather long, I've configured `numberOfRuns: 1` for my CI pipeline. However, if, and ideally only if, a performance assertion fails, it would be helpful if the test could be re-run for a given URL.

Increasing `numberOfRuns` isn't quite the same thing, as it will run multiple times also when the first test passes.

**Describe the solution you'd like**

An option to retry the assertions for a given page if the the first run fails.

**Describe alternatives you've considered**

Wrapping the entire LHCI run to re-run if it fails (this needlessly tests URLs again that have already passed).

Contributor guide

Open the contributing guide

Research direction

Start by tracing how Lighthouse CI runs a page URL, applies performance assertions, and uses numberOfRuns. Determine where a failed assertion could trigger a page-only retry without rerunning pages that already passed. Done means the retry option is defined, failed assertions retry as intended, and successful assertions do not cause extra runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
performance, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.