AnswerDotAI / AnswerDotAI/llms-txt
Multilingual sites handling + Tool listing proposal
- Dominant language
- Jupyter Notebook
- Stars
- 2.6k
- Forks
- 163
- Avg merge
- 1m
- Merged PRs (30d)
- 1
Description
## Question: How should llms.txt handle multilingual sites with thousands of localized pages?
While building a llms.txt generator (https://llmstxtgenerator.svcpool.com), I encountered a significant edge case with large multilingual websites.
**Example: www.lego.com**
- Contains thousands of pages across multiple languages
- See https://www.lego.com/magazine/sitemap.xml - hundreds of URLs for essentially the same content
- Patterns like `/en-us/page`, `/de-de/page`, `/fr-fr/page`, etc.
**The problem:**
Should the llms.txt standard:
1. **Include all localized versions** → results in massive files with duplicated semantic content
2. **Canonicalize URLs and include only one** → but which language? How to indicate alternatives?
3. **Add language metadata** to each entry → requires spec extension
**Current behavior:**
Most generators (including mine) process all URLs from sitemap, creating bloated llms.txt files where 90% is duplicate content in different languages.
**Proposed discussion:**
- Should the spec address canonical URL handling?
- Should there be a `lang` or `alternative-languages` field?
- Should generators automatically deduplicate based on URL patterns?
This affects scalability for enterprise sites. Would love the community's thoughts.
---
## Proposal: Add LLMs.txt Generator to Tools/Integrations Section
**Tool Name:** LLMs.txt Generator
**URL:** https://llmstxtgenerator.svcpool.com
**Type:** Online generator (no registration required)
### Description
I've built an LLMs.txt generator that handles websites of any size. Unlike other tools with restrictive page limits (10-50 pages) or registration requirements, this tool is designed for accessibility.
### Features
- Automatic website crawling and content extraction
- Multiple AI models (atm Google only, working on extending)
- Handles 5000 URLs per run (technical limit to keep servers healthy for all users)
- For larger sites: batch multiple runs or dedicated capacity available
- No registration or signup required
- No artificial restrictions
### Demo
Generated examples: https://llmstxtgenerator.svcpool.com/#example-output
### Request
Would you consider adding this tool to a "Tools" or "Integrations" section in the README? I believe it would help the community adopt the LLMs.txt standard more easily.
Happy to make any changes or provide additional information.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the llms.txt specification and README, then compare the multilingual URL examples and the proposed tool listing against the repository’s current guidance. Done requires a maintainer decision on localization handling and whether the generator belongs in a Tools or Integrations section, followed by any agreed documentation changes.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100