canonical / canonical/sphinx-stack

Link check rate limit causing delay

Open
#487 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
No language data
Stars
37
Forks
77
Avg merge
1d 5h
Merged PRs (30d)
3

Description

I've been asked by an engineer from a big project on what can cause a very long duration (>1 hour) of documentation testing.

It seems like most time is used by the link check test: [example](https://github.com/canonical/charmed-kubeflow-documentation/actions/runs/19890125455/job/57006627693?pr=26).

Tests duration:
Image

Timestamps for the link check:
Image

Looks like it could be that the rate limit is causing a 1-hour sleep.

I'm not sure what caused the rate limit, most likely - an abundance of links to GitHub resources, but I have an idea of what could be done to decrease the load during testing.

If we use the `sphinx-build` command to test links, we can limit its scope to specified files.

```shell
sphinx-build -b linkcheck srcdir outdir path/to/target1.rst path/to/target2.rst ...
```

See the [Run make linkcheck on only changed files](https://github.com/sphinx-doc/sphinx/issues/10643#top) Sphinx issue discussion for more context.

**We can limit it to only check the changed files in a PR**. That alone should improve the situation a lot.

As for checking the rest of the links (because they can become broken without changing in our docs) - we can do a full sweep on schedule, once a week, or month. We can even allocate weekdays and time slots to distribute it to avoid our line of products/repositories to avoid doing it at the same time (assuming that can create some problems).

I think this double approach is more reasonable than doing a full sweep for each commit to each PR with docs, given that we are experiencing rate limits.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.