canonical / canonical/starbase
Docs: Implement an alternative link checker
- Dominant language
- Makefile
- Stars
- 13
- Forks
- 6
- Avg merge
- 23h 56m
- Merged PRs (30d)
- 9
Description
In a recent [Rockcraft build](https://github.com/canonical/rockcraft/actions/runs/16355474142/job/46217812410?pr=921#step:6:5734), the sphinx-build link checker spent six hours trying to reach a link to [gnu.org](https://gnu.org). I've tried every timeout-related config setting available in the configuration, but none seem to stop it from infinitely looping.
I think it's caused by the FSF's recent work walling themselves off from [DDoSing by LLM agents](https://www.fsf.org/blogs/sysadmin/our-small-team-vs-millions-of-bots). Something about the response causes it to keep trying, possibly because of a limitation with [`Retry-After`](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-linkcheck_rate_limit_timeout). Or, Sphinx's user agent might be blocked now. Or, FSF is sending bots into a maze.
Regardless of what other domains are doing, it shouldn't bog our builds. If other savvy sites do likewise in the future, we don't want to be surprised by blocked builds.
Sphinx's link checker has always been slow and its capabilities meagre. I'd like to pursue other link checking solutions.
Contributor guide
Assessment
This issue has not been assessed yet.