AnswerDotAI / AnswerDotAI/llms-txt

Multilingual sites handling + Tool listing proposal

Open
#108 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.6k
Forks
163
Avg merge
1m
Merged PRs (30d)
1

Description

## Question: How should llms.txt handle multilingual sites with thousands of localized pages?

While building a llms.txt generator (https://llmstxtgenerator.svcpool.com), I encountered a significant edge case with large multilingual websites.

**Example: www.lego.com**
- Contains thousands of pages across multiple languages
- See https://www.lego.com/magazine/sitemap.xml - hundreds of URLs for essentially the same content
- Patterns like `/en-us/page`, `/de-de/page`, `/fr-fr/page`, etc.

**The problem:**
Should the llms.txt standard:

1. **Include all localized versions** → results in massive files with duplicated semantic content
2. **Canonicalize URLs and include only one** → but which language? How to indicate alternatives?
3. **Add language metadata** to each entry → requires spec extension

**Current behavior:**
Most generators (including mine) process all URLs from sitemap, creating bloated llms.txt files where 90% is duplicate content in different languages.

**Proposed discussion:**
- Should the spec address canonical URL handling?
- Should there be a `lang` or `alternative-languages` field?
- Should generators automatically deduplicate based on URL patterns?

This affects scalability for enterprise sites. Would love the community's thoughts.

---

## Proposal: Add LLMs.txt Generator to Tools/Integrations Section

**Tool Name:** LLMs.txt Generator
**URL:** https://llmstxtgenerator.svcpool.com
**Type:** Online generator (no registration required)

### Description
I've built an LLMs.txt generator that handles websites of any size. Unlike other tools with restrictive page limits (10-50 pages) or registration requirements, this tool is designed for accessibility.

### Features
- Automatic website crawling and content extraction
- Multiple AI models (atm Google only, working on extending)
- Handles 5000 URLs per run (technical limit to keep servers healthy for all users)
- For larger sites: batch multiple runs or dedicated capacity available
- No registration or signup required
- No artificial restrictions

### Demo
Generated examples: https://llmstxtgenerator.svcpool.com/#example-output

### Request
Would you consider adding this tool to a "Tools" or "Integrations" section in the README? I believe it would help the community adopt the LLMs.txt standard more easily.

Happy to make any changes or provide additional information.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the llms.txt specification and README, then compare the multilingual URL examples and the proposed tool listing against the repository’s current guidance. Done requires a maintainer decision on localization handling and whether the generator belongs in a Tools or Integrations section, followed by any agreed documentation changes.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.