QuantEcon / QuantEcon/quantecon-book-theme
Visual regression coverage gaps: threshold tolerance and missing RHS TOC snapshots
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 56
- Forks
- 24
- Avg merge
- 4h 33m
- Merged PRs (30d)
- 4
Description
Context
Surfaced while reviewing #388 (which addresses #387). The PR changed the LHS sidebar background colour and added a fade mask + hover-scroll to the RHS "On this page" TOC, yet all 45 Playwright visual regression tests passed without any baseline updates. That's not because nothing changed — the suite genuinely cannot see the changes it should.
Gap 1 — Threshold tolerance is too loose for colour shifts
playwright.config.ts currently sets:
```ts
toHaveScreenshot: {
maxDiffPixelRatio: 0.01, // 1% of pixels can differ
threshold: 0.2, // per-pixel: diff <20% RGB doesn't register
}
```
A pixel only counts as "different" if its RGB distance exceeds ~20%. That means:
#efefef→#ffffff≈ 6% per pixel — invisible to the comparator#efefef→rgba(239,239,239,0.4)over white ≈ ~2% per pixel — also invisible
So the entire sidebar surface changed colour and zero pixels registered as different.
The threshold: 0.2 setting is presumably tuned for cross-platform font/anti-aliasing tolerance (CI uses Ubuntu, local dev often macOS). Tightening it globally would produce noise on font-rendering deltas, but the current value lets through any colour shift below ~20% RGB distance — which is most realistic palette tweaks.
Possible directions
- Lower
thresholdselectively (e.g. 0.05 by default, override to 0.2 only for tests that include font-heavy regions). Would catch most palette shifts but stays tolerant of antialiasing where it matters. - Add explicit colour assertions (e.g.
await expect(sidebar).toHaveCSS('background-color', 'rgba(239, 239, 239, 0.4)')) for known-stable theme regions, complementing rather than replacing screenshot diffs. - Both — pixel-exact assertions where the value is known; screenshot diffs for everything else with a tighter threshold.
Gap 2 — RHS "On this page" TOC has no snapshot coverage
tests/visual/theme.spec.ts snapshots .qe-page__header, .qe-sidebar, full pages, code blocks, dark mode, math, toolbar, bold/italic typography — but nothing for .qe-page__toc. The main subject of PR #388 (fade mask, hover-to-scroll behaviour) is completely uncovered.
Additional sub-gap
None of the fixture pages exercise overflow scenarios:
```ts
const testPages = [
{ name: "homepage", path: "/index.html", hasPlots: false },
{ name: "intro", path: "/intro.html", hasPlots: false },
{ name: "getting-started", ... },
{ name: "python-by-example", ... },
{ name: "numpy", ... },
{ name: "matplotlib", ... },
];
```
All are short enough that the RHS TOC fits the viewport. The actual #387 bug (TOC clipping on long pages with many sections, e.g. prob_matrix.html) is not exercised by any test.
Possible directions
- Add a
.qe-page__tocsnapshot per existing page (mirrors-sidebar.png/-header.pngconvention). - Add a long-page fixture to the test set — either pin a deep lecture (
prob_matrix.htmlor similar) or build a synthetic page with many headings. - Add a hover-state snapshot to verify the mask removal + scrollbar appearance on hover.
Out of scope
This issue tracks the coverage gaps. The actual #387 fix lands in #388 as-is — no snapshot regeneration needed there since the comparator wouldn't flag a diff either way.
Acceptance
- Decision on threshold strategy (selective, explicit colour assertions, or both)
- RHS TOC snapshots added across existing fixture pages
- At least one long-page fixture covering RHS TOC overflow behaviour
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with playwright.config.ts and tests/visual/theme.spec.ts, then review the existing testPages and screenshot conventions. Decide the threshold strategy from the listed options, add RHS .qe-page__toc coverage including a long-page overflow case, and verify the acceptance items with the Playwright visual test suite.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- playwright, typescript
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100