Fetch llms.txt

It would be nice if you attempted to fetch llms.txt the same way you fetch robots.txt
It is still a proposal, but more and more sites are starting to adopt it.

https://llmstxt.org/

Start by locating how the reader fetches robots.txt, then review the llmstxt.org proposal linked in the issue. Done means the reader attempts to fetch llms.txt in the corresponding flow without disrupting existing URL processing.

Written by the indexing model from the issue text.