A content analysis of +500k llms.txt files

Sharing this here as it might be relevant for some folks:

At Common Crawl, we analyzed the contents of 584,107 llms.txt files from our July 2026 crawl.

TLDR: Two thirds are produced by a plugin, half follow the structure the specification defines, and a few files even contain prompt injections.

- Blog post: https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive
- Full report: https://huggingface.co/spaces/commoncrawl/llms.txt-report
- Dataset: https://huggingface.co/datasets/commoncrawl/llms.txt

Start with the linked Common Crawl blog post, full report, and dataset to understand the shared analysis of llms.txt files. No implementation target, named project file, or completion condition is stated, so the issue does not currently define what would count as done.

Written by the indexing model from the issue text.