AnswerDotAI / AnswerDotAI/llms-txt

Add support for sitemap / multiple llm.txt per domain

Open
#59 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.6k
Forks
163
Avg merge
1m
Merged PRs (30d)
1

Description

As far as I've noticed the current spec only allows one single file per domain. This makes it harder for tools that have multiple versions and control these via paths.

```
https://example.org # latest version
https://example.org/v1 # version 1
https://example.org/v2 # version 2
```

If the spec only allows a single llm.txt I only see two solutions right now:

1. Only create a llm.txt for the latest version
2. Merge all versions into one big llm.txt

Option 1 doesn't deliver great user experience if you're working with older versions, Option 2 increases tooling on the providers side and could lead into issues when the llm.txt grows so large that it doesn't fix into the context of the model. Additionally I am not sure how good the models could differentiate between the versions.

My suggestion is that there could be some kind of sitemap which tells the llm where it can find specific information.

I've yet have to come up with a great idea for a schema to suggest but wanted to get your feedback on the idea anyway.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the current llms.txt specification and its single-file-per-domain rule described in the issue. Compare the versioned URL examples and define a concrete sitemap or multiple-file schema, including how tools discover and select versions. Done means the proposal is specific enough to evaluate for inclusion in the specification.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.