AnswerDotAI / AnswerDotAI/llms-txt
Document compatibility fallback for linked .md resources
- Dominant language
- Jupyter Notebook
- Stars
- 2.6k
- Forks
- 163
- Avg merge
- 1m
- Merged PRs (30d)
- 1
Description
Hi! First, thanks for pushing this convention forward. I ran into a compatibility question while implementing `llms.txt` for a production storefront and wanted to share the findings.
The current guidance says that longer linked resources can be made available as Markdown by appending `.md` to the original URL, and the examples use paths like `.html.md`. That is semantically clear, but in manual compatibility testing we found that `.md` resources were less reliably fetchable by some major AI clients than equivalent `.txt` resources served as `text/plain`.
What we observed:
- Direct HTTP clients such as `curl` could fetch the `.md` resources normally.
- A normal browser could fetch the `.md` resources normally.
- Claude could fetch the `.md` resources.
- ChatGPT could fetch the same body when exposed through `.txt` + `text/plain`, but failed on the `.md`/Markdown surface in our tests.
- Gemini appeared to have trouble with `.md` resources, including unrelated `.md` files, but successfully fetched a `.txt` URL whose body was Markdown-like structured text.
This led us to choose `.txt` detail routes served as `text/plain` for the machine-readable resources linked from `llms.txt`, even though the body remains structured and Markdown-like. That seems to follow the practical goal of the convention, but it diverges from the `.md` detail-resource recommendation.
Would it make sense for the spec/docs to clarify one of these positions?
1. `.md` is the recommended semantic form, but `.txt` + `text/plain` is an acceptable compatibility fallback for linked resources.
2. Linked resources should prioritize content and content type over file extension, and `.md` is only a convention/example.
3. AI clients are expected to support `.md`/Markdown resources, in which case this may be a client-compatibility bug worth documenting separately.
I am happy to provide more detail if useful, but wanted to keep the issue focused on the extension/content-type guidance rather than one site's implementation.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the existing llms.txt guidance and its linked-resource examples, especially the recommendation to append .md. Review the reported .md versus .txt and text/plain compatibility findings, then clarify whether .txt is an acceptable fallback or whether content type should take priority; done when the documentation states the supported recommendation unambiguously.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- markdown
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100