QuantEcon / QuantEcon/quantecon-book-theme

Visual regression coverage gaps: threshold tolerance and missing RHS TOC snapshots

Open
#389 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
56
Forks
24
Avg merge
4h 33m
Merged PRs (30d)
4

Description

Context

Surfaced while reviewing #388 (which addresses #387). The PR changed the LHS sidebar background colour and added a fade mask + hover-scroll to the RHS "On this page" TOC, yet all 45 Playwright visual regression tests passed without any baseline updates. That's not because nothing changed — the suite genuinely cannot see the changes it should.

Gap 1 — Threshold tolerance is too loose for colour shifts

playwright.config.ts currently sets:

```ts
toHaveScreenshot: {
maxDiffPixelRatio: 0.01, // 1% of pixels can differ
threshold: 0.2, // per-pixel: diff <20% RGB doesn't register
}
```

A pixel only counts as "different" if its RGB distance exceeds ~20%. That means:

  • #efefef#ffffff ≈ 6% per pixel — invisible to the comparator
  • #efefefrgba(239,239,239,0.4) over white ≈ ~2% per pixel — also invisible

So the entire sidebar surface changed colour and zero pixels registered as different.

The threshold: 0.2 setting is presumably tuned for cross-platform font/anti-aliasing tolerance (CI uses Ubuntu, local dev often macOS). Tightening it globally would produce noise on font-rendering deltas, but the current value lets through any colour shift below ~20% RGB distance — which is most realistic palette tweaks.

Possible directions
  • Lower threshold selectively (e.g. 0.05 by default, override to 0.2 only for tests that include font-heavy regions). Would catch most palette shifts but stays tolerant of antialiasing where it matters.
  • Add explicit colour assertions (e.g. await expect(sidebar).toHaveCSS('background-color', 'rgba(239, 239, 239, 0.4)')) for known-stable theme regions, complementing rather than replacing screenshot diffs.
  • Both — pixel-exact assertions where the value is known; screenshot diffs for everything else with a tighter threshold.

Gap 2 — RHS "On this page" TOC has no snapshot coverage

tests/visual/theme.spec.ts snapshots .qe-page__header, .qe-sidebar, full pages, code blocks, dark mode, math, toolbar, bold/italic typography — but nothing for .qe-page__toc. The main subject of PR #388 (fade mask, hover-to-scroll behaviour) is completely uncovered.

Additional sub-gap

None of the fixture pages exercise overflow scenarios:

```ts
const testPages = [
{ name: "homepage", path: "/index.html", hasPlots: false },
{ name: "intro", path: "/intro.html", hasPlots: false },
{ name: "getting-started", ... },
{ name: "python-by-example", ... },
{ name: "numpy", ... },
{ name: "matplotlib", ... },
];
```

All are short enough that the RHS TOC fits the viewport. The actual #387 bug (TOC clipping on long pages with many sections, e.g. prob_matrix.html) is not exercised by any test.

Possible directions
  • Add a .qe-page__toc snapshot per existing page (mirrors -sidebar.png / -header.png convention).
  • Add a long-page fixture to the test set — either pin a deep lecture (prob_matrix.html or similar) or build a synthetic page with many headings.
  • Add a hover-state snapshot to verify the mask removal + scrollbar appearance on hover.

Out of scope

This issue tracks the coverage gaps. The actual #387 fix lands in #388 as-is — no snapshot regeneration needed there since the comparator wouldn't flag a diff either way.

Acceptance

  • Decision on threshold strategy (selective, explicit colour assertions, or both)
  • RHS TOC snapshots added across existing fixture pages
  • At least one long-page fixture covering RHS TOC overflow behaviour

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with playwright.config.ts and tests/visual/theme.spec.ts, then review the existing testPages and screenshot conventions. Decide the threshold strategy from the listed options, add RHS .qe-page__toc coverage including a long-page overflow case, and verify the acceptance items with the Playwright visual test suite.

Written by the indexing model from the issue text.

Assessment

Tech stack
playwright, typescript
Domain
testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.