What about crawlers?

Last night the Kiwix Wiki monitor was constantly throwing errors (Connection Timeout, 502).
The service did not restart but had apparently difficulties handling a high number of requests.
Peaking at live logs for a second, I see continuous requests from crawlers: Bytedance, Amazon, Claude, OpenAI, Bing were mentioned in this few-seconds window.

I added a denying robots.txt for both Wikis as there was none but I suppose crawlers don't look for it frequently (if they do at all). Nevertheless, about 30mn after that a successful monitor was seen.

Now that those things are more frequent, widespread and impacting our infrastructure, we might want to discuss what to do. Generalizing robots.txt seems in order. Should we do more?

The issue names no files, tests, or entry points. Start by reviewing the Kiwix Wiki monitor's live logs and the existing denying robots.txt changes, then define the crawler-handling approach and how its effect on timeouts and 502 errors will be measured.

Written by the indexing model from the issue text.