cloudfour / cloudfour/lighthouse-parade
Add a Lighthouse integration test against a local fixture page
- Dominant language
- TypeScript
- Stars
- 373
- Forks
- 15
- Avg merge
- 13h 36m
- Merged PRs (30d)
- 34
Description
## Background
`src/lighthouse.ts` is at **17.5% coverage, with 0% of its functions executed**. No test ever runs the real `runLighthouseReport` — `test/scan-task.test.ts` injects a fake via the `lighthouse` option, and `src/crawl.mock.ts` does the same for the crawler. That dependency injection is good design for testing orchestration, but it means the actual Lighthouse integration is never exercised.
The consequence showed up while triaging the Lighthouse 13 upgrade (#304). That PR fails CI, but **it fails by accident rather than by test design**. `src/lighthouse.ts` calls `require.resolve('lighthouse/lighthouse-cli')` at module scope, and `src/scan-task.ts` imports that module for a default parameter — so the import throws before any test body runs. Move that `require.resolve` inside the function (a refactor anyone might reasonably make) and #304 would report fully green while being completely broken.
## Proposal
Add an integration test that spawns the real Lighthouse CLI against a **local static fixture page**, not a live URL. Serving our own HTML keeps it deterministic and offline; pointing it at a real site would be flaky.
Assert on **structure only**:
- The subprocess exits 0.
- The output parses as CSV.
- The expected columns are present (`category`, `name`, `title`, `type`, `score`).
- Score values parse as numbers within 0–1.
Do **not** assert on specific scores. Lighthouse results are nondeterministic and vary run to run on identical input — that would be a flaky test, not a useful one.
## Running it
This needs Chrome and takes real time, so it shouldn't run on every PR. Options worth weighing:
- A separate vitest project or a `test:integration` script, run nightly on a schedule plus manually via `workflow_dispatch`.
- Or gate it behind an env var so it's opt-in locally and scheduled in CI.
## Why it matters
This is the only thing that would meaningfully de-risk future Lighthouse majors. Right now a Lighthouse upgrade can only be validated by hand.
Related: #304
Contributor guide
Research direction
Start by reading src/lighthouse.ts, src/scan-task.ts, and test/scan-task.test.ts to understand the real Lighthouse invocation and existing dependency injection. Add a local fixture-page integration test that checks subprocess success, CSV structure, expected columns, and numeric 0–1 scores, then run it through the chosen opt-in or scheduled CI path without asserting exact scores.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- ci-cd, cli, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100