canonical / canonical/sphinx-stack

Linkcheck runtime improvement

Open
#567 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
37
Forks
77
Avg merge
1d 5h
Merged PRs (30d)
3

Description

During my experiments with AI, I was able to wring out a genuine improvement to the linkchecker runtime, with a simple configuration change. In my docs (which are large, and have well over a thousand links to check), the linkchecker takes about 10 minutes with the default settings currently in the SP:

```python
# give linkcheck multiple tries on failure
# linkcheck_timeout = 30
linkcheck_retries = 3
```

If the settings are changed instead to:

```python
# Give linkcheck multiple tries on failure
linkcheck_timeout = 15
linkcheck_retries = 2

# Number of parallel workers for linkcheck (default is 5)
# Higher values work well for network I/O-bound tasks
linkcheck_workers = 20
```

I cut down the runtime to 1.5 minutes (over 80% improvement).

I'm leaving this here as an issue rather than a PR because I'm not sure how the workers setting would impact other projects - the parallelism works well in the Server docs, but I could see a case for not including it in the SP if a project contains a high volume of links to a specific site, which could lead to linkcheck failures due to rate limiting. This part needs more general testing (but I recommend the timeout and retries both be reduced for the general performance improvement).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.