Audit the exclude list: valid links are being silenced instead of verified

lychee.toml exclude has grown to 60 entries, and I think only about half of them are doing the job the key is named for. The rest are valid, reachable links that the checker or the host got wrong, parked in exclude because that was the only lever available at the time. PR #2 is the most recent example: four URL shapes that resolve fine, excluded because nothing else in the pipeline could absorb them.

The cost is that exclude is unconditional, permanent, and silent. A URL in that list is never checked again for anything, so the day the target 404s, nobody hears about it.

What is actually in the list

Rough triage of the current 60 entries by the problem each one is solving:

So roughly 30 of 60 entries are links that work.

The redirect class already has two other homes, and exclude is the worst of the three

A 3xx is not an error in this pipeline. accept deliberately leaves 3xx out so redirects land in .redirect_map and get their own ## Redirects section, which is informational, not a failure. On top of that, generate-link-report.sh already carries a skip_redirect() function (line 408) that suppresses uninteresting redirects by shape: trailing slash, query-string-only, locale, auth, version-pointer, plus hardcoded per-host cases for agentgateway.dev/examples/ and sigs.k8s.io/gateway-api.

That means the same problem is being solved in three places with three different mechanisms, and the 11 entries that ended up in exclude got the strictest one. ^https?://github\.com/[^/]+/[^/]+/releases/latest$ is the clearest case: the reason it is excluded is "it redirects," but the effect is that a deleted or renamed repo behind that URL is now permanently unreportable across every product.

Specific entries I would push back on

Not arguing these were wrong to add, just that they are broader than their comments read:

• ".*127\\..*" matches 127. anywhere in a URL, path and query included, not just the loopback host. It is also largely redundant with exclude_all_private = true.
• ".*\\.json" drops every JSON URL in every product. A genuinely broken link to a schema or config file is invisible.
• "https://www.linkedin.com/*" reads like a path glob but is a regex, where /* means zero-or-more slashes. It excludes the whole domain. The . are unescaped too.
• "http://agentgateway-enterprise-controller.kagent*" has t* as zero-or-more t, is unanchored, and is http-only.
• ".*:8200.*" and ".*:9080.*" match those digits anywhere in the URL.
• ".*storage\\.googleapis.*" excludes an entire CDN to work around one host's checker behavior.

What I would propose

The repo's own stated preference is already the right one. generate-link-report.sh says of verify-anchors.sh that it "checks them for real instead of blanket-ignoring the URL." I would make that the rule rather than the exception, and shrink exclude back to "not a link."

1. Move the blocked-host class into curl-retry-patterns.txt. That file is built for stable bot-blocks and odd status codes, and a real 404 still fails curl -sSf, so the link stays checked. This is a file move plus a verification run, not new code.
2. Test the four JS-anchor entries against verify-anchors.sh. If it resolves them, delete them. If it does not, the comment on each should say why, which is more useful than the current "lychee can't see it."
3. Add an accepted-redirects list that filters the Redirects section, not the check. A canonical short form would still be fetched, still 404-checked, and simply not listed as a redirect when the final URL returns 200. Folding the hardcoded skip_redirect() host cases into the same list at the same time would put all redirect policy in one reviewable file instead of split between a TOML array and a bash function.
4. Add an audit script. Point it at built public/ trees across products and have it report, per exclude entry: how many URLs it matches (zero means dead weight), how many of those now pass a plain curl (exclusion no longer needed), and whether it matches more hosts than the comment claims. This is the part that makes the list maintainable instead of append-only. PR #2 did this audit by hand for two patterns and it was the most convincing part of the PR.
5. Write down the bar for new entries, matching what PR #2 volunteered: what it matches, what it leaves checked, measured against a real URL corpus. A short checklist in the repo, or a PR template.

Reasons this could be the wrong call

Worth stating, because I am not confident the whole thing is worth doing:

• The exclude list is free at runtime. Every alternative above costs CI wall clock. curl-retry-failures.sh already runs on a 600s budget and logs URLs it cuts off, so moving a dozen hosts into it may push real failures past the budget and manufacture the false positives the script exists to remove. Steps 1 and 3 should be measured against that budget before landing.
• The risk each exclude hides is small in practice. A dead discord.gg invite or a renamed repo is a real but low-frequency problem, and the weekly report already runs long.
• A defensible smaller version of this is steps 4 and 5 alone: keep the list, but make it auditable and hold new entries to the PR #2 bar. That fixes the growth problem without touching the pipeline.
• The classification above is mine and some entries are arguable. twitter.com/soloio_inc and slack.solo.io are closer to policy decisions than to checker bugs, and I put .*articles/4414409064596.* in "content team decision" purely because the comment says so.

Related: #4, which is the same problem reaching this list from generated pages.

Start by reading lychee.toml, curl-retry-patterns.txt, curl-retry-failures.sh, verify-anchors.sh, and generate-link-report.sh, then run the existing checks against built public/ trees. Compare the proposed exclude classifications with measured matches, plain-curl results, anchor verification, and the 600-second retry budget. Done means the agreed scope is implemented, redirect and exclusion policy is reviewable, and the audit/checklist output supports future entries.