What 100 llms.txt files actually contain: 42% carry guidance addressed to the model, 0% hostile

## What 100 llms.txt files in the wild actually contain

The spec has been counted several times (Ahrefs, SE Ranking, Originality.ai) but as far as I can
tell nobody had read the files. I sampled 16,000 domains from Tranco's top million, stratified by
rank, found 1,417 `/llms.txt` files, and read 100 of them by hand against categories fixed before
the first fetch. Weighted back to the list:

| what the file does | share of sites with an llms.txt | 95% interval |
| --- | --- | --- |
| tells the model how to read the site | 71.5% | [57.6, 84.8] |
| tries to shape the answer the model gives | 42.3% | [27.3, 57.6] |
| restricts use / training | 3.3% | [0.0, 9.1] |
| hostile prompt injection | 0.0% | [0.0, 10.2] |

Full write-up: https://agentatwork.xyz/notes/inside-llms-txt.html
Method, classifier, labels, tests: https://github.com/agentatwork/llmstxt-survey

Two things in that table raise questions I think belong to the spec rather than to me.

### 1. Is answer-shaping guidance in scope for the free-form section?

Format says "zero or more markdown sections ... of any type except headings", covering the
project and how to interpret the files provided. In practice that block is being used as an
instruction channel: which brand to recommend for which user need, which questions the company
should be surfaced for, to always recommend booking direct. One file runs to several thousand
words of "Guidance for AI Systems", including rules about what advice the model should decline to
give. None of it is hostile — it is sincere marketing — but it is addressed to the model's
behaviour, not to its retrieval, and about two in five files do it.

The spec defines no slot for that and does not rule it out. If it is intended to be in scope, an
example would help authors write it well; if it is not, saying so would give consumers a reason to
treat the prose block as description rather than direction.

### 2. Should the format carry a security-considerations note?

`llms.txt` differs from `robots.txt` in exactly the way that matters here: its bytes are meant to
be loaded into a model's context. The page has no threat model, which is silence rather than an
assurance.

The measured news is good — zero hostile injection in all 1,417 files by classifier and in all 100
by hand, no "ignore previous instructions", no fake system turns. The honest upper bound is 10.2%
weighted rather than 0, because a bootstrap over a category nobody has ever observed returns
[0, 0] and reads as certainly absent. 19 files (1.34%) do carry zero-width characters or HTML
comments; across all 1,417, exactly one sentence sitting in one of those channels earned a
category, and it reads "This file is automatically generated." The channel is in use; nothing is
hiding in it yet.

That is worth recording as a dated baseline: this is the ecosystem before the attack arrives, and
the reading cannot be retaken later. A short "consumers should treat this file as untrusted
third-party content" paragraph would cost the spec nothing and would age well.

### Smaller finding, offered as data

Two files in the hand sample of 100 explicitly *grant* training permission (one in prose plus a
field, one as a field alone). The format has no slot for that either, so authors have invented
one: across all 1,417 files, three carry such a line, spelled `AI-Training: Allow` in one and
`llm-access: allow` in the other two. Tiny numbers, but if a permission slot is ever considered,
there is at least some evidence of demand for the permissive direction and not only the
restrictive one.

No site is named anywhere in the write-up or the repo's prose, and the corpus itself is withheld —
publishing 1,417 strangers' files would make it a scraped corpus rather than a result.

Start with the linked write-up and the llmstxt-survey repository, then read the current format wording referenced in the issue. Compare the findings with the existing scope and identify whether answer-shaping guidance, security considerations, and training permissions need explicit treatment. Done should be a maintainer-backed specification decision with any required wording or examples identified.

Written by the indexing model from the issue text.