solo-io / solo-io/docs-theme-extras

selectedPages() is never called, so 17 specs silently no-op for any consumer without [[pages]]

Open
#49 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
0
Forks
0
Avg merge
18h 13m
Merged PRs (30d)
20

Description

Summary

selectedPages() in tests/helpers/crawl.ts exists to fall back to a crawl of builtRoot when a consumer declares no [[pages]] entries. It is exported, documented, and never called by anything:

$ grep -rn "selectedPages" . --exclude-dir=node_modules
tests/helpers/crawl.ts:56:export function selectedPages(): CrawledPage[] {

Its own comment names the specs that are supposed to use it:

Return only the explicitly listed pages from CONFIG, falling back to the crawl when no [[pages]] entries are declared. Specs that test specific behaviors against representative pages (contrast, viewport, browser specs) use this; specs that scan everything (content, static) call crawlBuiltRoot.

All three named specs read target.pages raw instead. So do 14 others — 17 in total:

alert-body, back-to-top, brand, browser, contrast, cross-browser, mermaid-render, mobile-drawer, openapi-example-contrast, openapi-method-colors, page-feedback, reuse-image-dark-pair, sidebar-rail, theme-toggle, toc-layout, version-banner-link, viewport.

Impact

For a consumer with no [[pages]] in its config, target.pages is [] (config.ts:295 maps over pagesData ?? []). Every one of those 17 specs then either hits a skip guard or runs against an empty sample. The skip is worded as a benign configuration note, so it reads as "not applicable here" rather than "this check is off":

// tests/alert-body.spec.ts
const SAMPLE = target.pages.filter((p) => target.versionOf(p.url) !== null);
const PAGE = SAMPLE[0]?.url ?? null;
test.skip(PAGE === null, "no non-landing pages configured");

kgateway.dev is in exactly this state — its .docs-test.toml declares zero [[pages]], so all 17 have been skipping since the harness was adopted there. I would guess other consumers are too; the config is easy to adopt without the optional block.

The fallback was clearly written to prevent precisely this. It just never got wired in.

Suggested fix

Swap the raw reads for selectedPages() in the specs whose comments already say they use it, at minimum the three it names. That makes the no-[[pages]] case degrade to "sample from the crawl" instead of "silently do nothing", and it needs no consumer-side change.

Worth considering alongside it: a guard that fails when a spec's page sample comes back empty, in the spirit of tests/scan-roots.spec.ts treating a scan root that reads nothing as an error rather than a pass. A skip that means "the harness found no input" and a skip that means "this consumer opted out" should not look the same in the report.

How this surfaced

Found while reviewing kgateway-dev/kgateway.dev#978, which fixes a related class of silent skip: .docs-test.toml had drifted from versions.json, so the per-version specs never checked 2.3.x and the version dropdown spec chased a 2.0.x tree that no longer builds.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/helpers/crawl.ts and the three specs named in its comment, then trace config.ts:295 to understand the empty target.pages case. Review the listed specs for raw page reads and compare their behavior with selectedPages(). Done means consumers without [[pages]] obtain crawled sample pages instead of silently skipping or testing an empty sample.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing-qa, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.