thefrontside / thefrontside/staticalize
Support extra crawl paths beyond what's in the sitemap
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 8
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Problem
Right now staticalize's crawl surface is exactly the URLs listed in /sitemap.xml, plus links it discovers by following anchors/assets from those pages. There's no way to say "also crawl these paths that aren't in the sitemap."
The concrete case that led to this: I have a handful of legacy pages I want in the static output for backward compatibility (people still have old bookmarks), but I deliberately keep them out of /sitemap.xml so search engines don't index them. Combined with <meta name=\"robots\" content=\"noindex\"> and a robots.txt Disallow, this is the standard "preserved but unindexed" pattern.
Because staticalize crawls only the sitemap, these pages aren't produced in the output — even though they have real routes on the running server. My current workaround is a shell step after `staticalize`:
```sh
staticalize http://localhost:8005 --output=built --base=https://example.com \
&& cp -R legacy/* built/
```
This works, but it bypasses the actual serving code path (I lose the same headers, HTML transforms, and asset-following behavior staticalize gives everything else).
Proposed
A repeatable flag for additional paths to enqueue:
```sh
staticalize http://localhost:8005 \
--output=built \
--base=https://example.com \
--path /hat.html \
--path /checked-vs-unchecked.html
```
Each --path gets treated exactly like a sitemap entry: fetched with staticalize's downloader, has its links followed to pull in referenced images/CSS/JS, and gets written to the corresponding location in --output. But it's not derived from /sitemap.xml, so the public sitemap doesn't have to list it.
Alternatives considered
- Separate "internal" sitemap URL (e.g.
--sitemap /sitemap-internal.xml) that staticalize merges with the primary. More XML plumbing for the app, but nice if you have many paths. --paths-from <file>taking a newline-delimited list. Convenient for large sets.
--path (repeatable) probably covers 90% of cases with minimal API surface.
Context
Used from a Deno project (@cowboyd/cogentdude.com) that deploys to GitHub Pages via Actions. Happy to open a PR if the direction sounds reasonable.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the CLI option parsing and the sitemap enqueueing flow, then trace how queued URLs are downloaded and written to the output. Add repeatable path handling without requiring those paths in the sitemap, preserving the existing link-following behavior, and verify that each supplied path is generated in the corresponding output location.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cli, web-dev
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100