kestra-io / kestra-io/docs

Lighthouse benchmark reports false regressions: 11 of 18 sampled pages are measured once

Open Beginner friendly
#5,707 0 comments 0 reactions 0 assignees View on GitHub
area/devops area/qa kind/cooldown kind/performance
Dominant language
Astro
Stars
155
Forks
177
Avg merge
1d 20m
Merged PRs (30d)
192

Description

## What happens

The PR benchmark flags score regressions on pages that did not change. Reviewers learn to ignore the 🔻 markers, which is the failure mode that makes the whole check worthless.

## Evidence: two runs, same bytes

PR #5705 ran the benchmark twice, on heads `ea30011` and `441cc12`. The only difference between those commits is a single trailing newline added to `scripts/visual-diff-report.mjs`, a script that is never executed at build time. The deployed site is byte-identical between the two runs.

- Run A: [35347231169](https://github.com/kestra-io/docs/actions/runs/35347231169), shard CPU index 2937 / 2489 / 2492 / 3002, deltas hidden on 8 pages
- Run B: [35348593620](https://github.com/kestra-io/docs/actions/runs/35348593620), shard CPU index 2484 / 3021 / 2457 / 2974, deltas hidden on 9 pages

Performance scores, grouped by the `runs` count each page carries in `tests/fixtures/page-sample.mjs`:

| Page | `runs` | Run A | Run B | Swing |
|---|---|---|---|---|
| Get Started | 1 | 89 | **70 🔻 -11** | 19 |
| Plugin Debug Return Page | 1 | 94 | **72 🔻 -19** | 22 |
| Blueprint Audit Logs CSV Export | 1 | 86 | 94 | 8 |
| Blog Post (sample) | 1 | 87 | 94 | 7 |
| Cloud | 1 | 90 | 94 | 4 |
| VS Page (sample) | 1 | 95 | 99 | 4 |
| Flow (full featured docs) | 3 | 97 | 97 | 0 |
| Blueprints Landing | 3 | 82 | 81 | 1 |
| Plugins Landing | 3 | 98 | 94 | 4 |

Aggregated across all 18 pages:

| `runs` | pages | mean swing | max swing |
|---|---|---|---|
| 1 | 12 | 6.1 | 22 |
| 3 | 4 | **1.3** | **4** |
| 5 | 2 | 6.5 | 10 |

Both pages that flagged a regression in run B sit in the `runs: 1` group. `SCORE_THRESHOLD` is 10 points (`scripts/lighthouse-shared.mjs:53`), and single-run noise clears it comfortably.

## Root cause

`runsFor()` in `scripts/lighthouse-shared.mjs:119` returns 1 for any page without an explicit `runs` value:

```js
export function runsFor(page, override = 0) {
if (!page.runs || page.runs < 2) return 1
return override || page.runs
}
```

11 of the 18 entries in `PAGES` have no `runs` key, so they get one Lighthouse pass and no median. `runMedian()` exists and works, it is just not applied to most of the sample.

The `benchmarkIndex` guard (`BENCHMARK_INDEX_THRESHOLD`, 10%) does not cover this. It compares each shard's median CPU index against the baseline's and hides that page's deltas when the gap is too wide. In run B two shards landed at 2484 and 2457 against a 2442 baseline, inside 2%, so their pages were compared at face value. The guard correctly protects against a slow runner; it does nothing about run-to-run variance on a comparable one.

The `runs: 5` row above looks contradictory, but its two pages are Home (swing 3) and Docs Landing (swing 10), and Docs Landing sat on a suppressed shard in one of the runs. Two data points, not a trend.

## Options

**1. Give every page `runs: 3`.** Cheapest to reason about, and the data says a median of 3 already cuts the mean swing from 6.1 to 1.3 points. Cost: total Lighthouse passes go from 34 to 54, roughly 60% more measuring time, spread over the existing 4 shards. Shard jobs currently finish in 4 to 5 minutes, so expect 7 to 8.
*Critical failure mode:* it buys a constant factor, not a guarantee. A page that is genuinely bimodal (an SSR page racing a cold worker, say) still flips, and at 3 runs the median can land on either mode.

**2. Require a regression to reproduce before marking it.** Keep single runs, but only render 🔻 when the same page regresses against the baseline on two consecutive commits. Nearly free in runner time.
*Critical failure mode:* it delays every true regression by one commit, and gives no signal at all on a single-commit PR, which is most of them.

**3. Widen the thresholds per page based on observed variance.** Record each page's historical spread in the baseline artifact and scale `SCORE_THRESHOLD` per page instead of using a flat 10.
*Critical failure mode:* the most correct and the most machinery. It needs several baseline runs before the numbers mean anything, and a noisy page quietly acquires a threshold so wide that a real 15-point regression on it never reports.

Option 1 is the recommendation: it targets the measured cause, the runner cost is known, and it needs no new state. Options 2 and 3 are worth revisiting if 3 runs turns out not to be enough.

## Separate observation: About Us

About Us scored **84 in both runs**, flagged 🔻 -14 against a baseline of 98 each time. It carries `runs: 3`, and an identical median across two runs on different shards is not the noise pattern above. Either main genuinely regressed on that page and the baseline artifact predates it, or the baseline is stale for another reason. Worth its own look; it is not explained by this issue.

Contributor guide

Open the contributing guide

Research direction

Start with runsFor() in scripts/lighthouse-shared.mjs and the page definitions in tests/fixtures/page-sample.mjs. Review how runMedian() is used by the benchmark, then make pages without an explicit runs value use three runs as recommended. Done means the affected pages receive median-based measurements and the benchmark no longer reports the demonstrated single-run swings.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
performance, testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.