GoogleChrome / GoogleChrome/lighthouse

Ability to automatically run several "rounds" of the Lighthouse test in DevTools and get an averaged result

Open
#14,421 3 comments 0 reactions 1 assignee Claimed by @connorjclark View on GitHub
DevTools needs-design P2
Dominant language
JavaScript
Stars
30.8k
Forks
9.8k
Avg merge
1d 14h
Merged PRs (30d)
20

Description

**Feature request summary**
Ability to automatically run several "rounds" of the Lighthouse test and get an averaged result (plus access to data on the individual rounds so you can see how much variation there is).

The variability guide [already recommends](https://github.com/GoogleChrome/lighthouse/blob/main/docs/variability.md#run-lighthouse-multiple-times) that users do this - but it would be great to add this as an option in DevTools.

**What is the motivation or use case for changing this?**
Lighthouse test results can vary quite a bit, even when running them one after the other. I think *this is especially the case for pages that have ads*, because sometimes different types of ads (e.g. animated/video vs static/image) will be loaded which affects the performance score (not just the Publisher Ads plugin score).

**How is this beneficial to Lighthouse?**
The goal of Lighthouse is to help developers test performance improvements that they've made to their sites/apps. In a recent test I ran Lighthouse 4 times, and got these scores: 60, 62, 58, 56. With such large variation in the scores, it's possible that a new performance improvement that I added to my app isn't captured in the score after one test - the large variation could end up convincing me that I've actually *decreased* performance. It's only after averaging over a few tests that it's clear that performance was improved. Manually running several tests and aggregating the results is time-consuming compared to what could be a simple button click.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.