AnswerDotAI / AnswerDotAI/llms-txt

Document compatibility fallback for linked .md resources

Open Beginner friendly
#130 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.6k
Forks
163
Avg merge
1m
Merged PRs (30d)
1

Description

Hi! First, thanks for pushing this convention forward. I ran into a compatibility question while implementing `llms.txt` for a production storefront and wanted to share the findings.

The current guidance says that longer linked resources can be made available as Markdown by appending `.md` to the original URL, and the examples use paths like `.html.md`. That is semantically clear, but in manual compatibility testing we found that `.md` resources were less reliably fetchable by some major AI clients than equivalent `.txt` resources served as `text/plain`.

What we observed:

- Direct HTTP clients such as `curl` could fetch the `.md` resources normally.
- A normal browser could fetch the `.md` resources normally.
- Claude could fetch the `.md` resources.
- ChatGPT could fetch the same body when exposed through `.txt` + `text/plain`, but failed on the `.md`/Markdown surface in our tests.
- Gemini appeared to have trouble with `.md` resources, including unrelated `.md` files, but successfully fetched a `.txt` URL whose body was Markdown-like structured text.

This led us to choose `.txt` detail routes served as `text/plain` for the machine-readable resources linked from `llms.txt`, even though the body remains structured and Markdown-like. That seems to follow the practical goal of the convention, but it diverges from the `.md` detail-resource recommendation.

Would it make sense for the spec/docs to clarify one of these positions?

1. `.md` is the recommended semantic form, but `.txt` + `text/plain` is an acceptable compatibility fallback for linked resources.
2. Linked resources should prioritize content and content type over file extension, and `.md` is only a convention/example.
3. AI clients are expected to support `.md`/Markdown resources, in which case this may be a client-compatibility bug worth documenting separately.

I am happy to provide more detail if useful, but wanted to keep the issue focused on the extension/content-type guidance rather than one site's implementation.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the existing llms.txt guidance and its linked-resource examples, especially the recommendation to append .md. Review the reported .md versus .txt and text/plain compatibility findings, then clarify whether .txt is an acceptable fallback or whether content type should take priority; done when the documentation states the supported recommendation unambiguously.

Written by the indexing model from the issue text.

Assessment

Tech stack
markdown
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.