Proposal: `llms-tldr.txt` - Token-Efficient Knowledge Summary

# Proposal: `llms-tldr.txt` - Token-Efficient Knowledge Summary

## Summary

Add an optional third file to the [llms.txt](https://llmstxt.org/) family: **`/llms-tldr.txt`**.

It delivers **substantive site knowledge in one HTTP request**, within a declared token budget — without replacing `llms.txt` (navigation) or `llms-full.txt` (exhaustive listings).

**Reference implementation:** Cybermaps WordPress Plugin **v3.1.9** (protocol **`llms-tldr.txt v1.1`**).

---

## Problem

| File | Strength | Failure mode on large sites |
|---|---|---|
| `llms.txt` | Fast orientation (~20 links + YAML) | Agent must crawl each URL for real knowledge |
| `llms-full.txt` | Deep per-post text (~100+ entries) | Token count explodes (500+ posts → millions of tokens) |

Modern models commonly offer **128K–200K** context windows. A multi-million-token `llms-full.txt` is truncated, skimmed, or processed with degraded citation quality.

**Gap:** a single, budget-bounded file that is **more than links** and **less than everything**.

---

## Proposed Solution

**Path:** `/llms-tldr.txt` at site root (same discovery pattern as `llms.txt`).

**MIME:** `text/plain; charset=utf-8`

**Body:** Markdown structured for machine ingestion (headers + bullet metadata, not long prose).

**Protocol header (v1.1, recommended):**

```text
# {Site Name} | Knowledge TL;DR
> Protocol: llms-tldr.txt v1.1
> Generated-by: {generator name}
> Generated-at: {ISO8601 UTC}
> Token-estimate: {integer}
> License: {optional, e.g. CC-BY-4.0}
```

---

## Generation Pipeline (Normative Stages)

Generators MAY implement additional logic; these stages define interoperable expectations.

### Stage 1 — Pinned content (optional)

Human-selected URLs or post IDs included first, bypassing automated filters. For pricing, policies, and docs the operator knows agents must see.

### Stage 2 — Candidate pool

Fetch a bounded set of recent content (implementations often default to **100** posts from configurable types). Respect site-wide and per-item exclusion flags consistent with `llms.txt`.

### Stage 3 — Quality filtering

Exclude items that are:

- Empty or shortcode-only (no readable text)
- Demo/placeholder (e.g. "Lorem ipsum", "Sample page", "Hello world")
- Below a minimum substantive length (reference: **50 characters**)

### Stage 4 — Scoring, deduplication, and layout

**Scoring:** rank remaining items with explainable signals (reference implementation uses AI metadata: info gain, depth, entities, plus optional substance/structure bonuses). Drop items below a configurable threshold (reference default: **0.6**).

**Deduplication:** if two items share **≥2** taxonomy term slugs (categories or tags), treat as one topic cluster; keep the **highest-scoring** item. Optionally emit a cluster note: `> N posts in this cluster. Top-scoring entry shown.`

**Layout (hybrid, recommended):**

- **Section headers** by primary topic label (reference: primary category, else first tag, else `General`) — e.g. `## Topic: WordPress Development`
- **Intent** (informational, transactional, …) as **per-entry metadata**, not necessarily as the only sectioning model — preserves compatibility with intent-aware `llms.txt` and search APIs

### Stage 5 — Token budgeting

Stop adding automated entries when estimated tokens would exceed a configurable cap (reference default: **80,000**). Pinned content is typically exempt from the cap. If truncated, append a machine-readable footer listing omitted topic labels.

**Token estimate:** implementations MAY use `strlen(content) / 4` or another documented heuristic; expose the estimate in the protocol header when possible.

---

## Entry Format (Reference)

```markdown
## Topic: WordPress Development
> 12 posts in this cluster. Top-scoring entry shown.

### Building Custom Gutenberg Blocks
- **URL**: https://example.com/gutenberg-blocks/
- **Core Intelligence**: {dense snippet — entities, outcomes, or excerpt}
- **Topic**: WordPress Development
- **Intent**: informational
- **Score**: 0.84
- **Key Entities**: Gutenberg, block-editor, React
```

| Field | Required | Notes |
|---|---|---|
| `###` title | Yes | Post or resource title |
| **URL** | Yes | Canonical URL |
| **Core Intelligence** | Yes | Primary knowledge payload |
| **Topic** | Recommended | Section/taxonomy label |
| **Intent** | Recommended | Agent routing (aligned with `llms.txt` silos) |
| **Score** | Optional | Generator confidence / density (0.0–1.0) |
| **Key Entities** | Optional | Comma-separated when extractable |

---

## Discovery & Caching

- **Discovery:** link from `llms.txt` Optional section (e.g. `tldr: /llms-tldr.txt`)
- **Caching:** recommend **1-hour** HTTP or application cache with invalidation on content change
- **Static mirror:** optional on-disk copy for zero-PHP delivery (reference: Cybermaps Static File Engine)

---

## Relationship to Other Files

| File | Role |
|---|---|
| `llms.txt` | Structure, links, YAML sitemap, intent silos for navigation |
| `llms-tldr.txt` | Budgeted knowledge extract for one-shot ingestion |
| `llms-full.txt` | Exhaustive listings when token budget allows |

None replaces the others.

---

## Reference Results (Cybermaps Test Site)

~150 posts, 8 categories:

| | `llms-full.txt` | `llms-tldr.txt` |
|---|---|---|
| Estimated tokens | ~450,000 | ~28,000 |
| Distinct topics represented | All posts (diluted) | ~14 clusters |
| Quality filtering | No | Yes |
| Fits 128K window | No | Yes |

Figures vary by site; the pattern — **full vs. budgeted** — is the point.

---

## Alternatives Considered

| Approach | Why not sufficient alone |
|---|---|
| Vector DB / RAG | Multiple round-trips; loses single-file simplicity of llms.txt |
| Chunking `llms-full.txt` | Agent state across chunks; per-chunk attention still limited |
| `llms.txt` only | Requires per-URL crawl for substance |

---

## Reference Implementation (Cybermaps v3.1.9)

| Item | Location |
|---|---|
| HTTP handler | `src/Discovery/LLMSTLDR.php` |
| Pipeline | `src/Discovery/LLMSTLDR/LLMSTLDRGenerator.php` |
| Quality | `LLMSTLDRQuality.php` |
| Scoring | `LLMSTLDRScorer.php` |
| Clustering / dedup | `LLMSTLDRCluster.php` |
| Rendering | `LLMSTLDRRenderer.php` |
| Token budget | `TokenBudget.php` |
| Transient cache | `cybermaps_tldr_cache` (1h) |
| REST mirror | `GET /wp-json/cybermaps/v1/llms-tldr` |
| Multilingual static | `{lang}/llms-tldr.txt` when multilingual hub enabled |

**Further reading:**

- [Technical deep-dive](https://cybermaps.dev/tech/llms-tldr-whitepaper/)
- [Blog post / rationale](https://cybermaps.dev/blog/llms-tldr/)

Scoring, filters, and dedup rules are plain heuristics — no external API required to reproduce the idea.

---

## Specification Request

Consider adding **`/llms-tldr.txt`** as an **optional third file** in the llms.txt specification:

1. Optional protocol header block (`v1.1` fields above)
2. Five-stage generation model (pinned → pool → quality → score/dedup/layout → token budget)
3. Hybrid layout guidance (taxonomy sections + per-entry intent)
4. Recommended default token cap (**80,000**, generator-configurable)
5. Link from `llms.txt` Optional section

Cybermaps provides a working, GPL-licensed reference. Feedback on field names, dedup rules (2+ shared terms), and header semantics is welcome.

Start by reading the existing llms.txt specification and comparing its optional section with the proposed /llms-tldr.txt discovery, header, pipeline, and budgeting requirements. Review the referenced Cybermaps implementation files, including src/Discovery/LLMSTLDR.php and the generator, quality, scorer, cluster, renderer, and TokenBudget classes; done requires a maintainer-approved specification change with clear interoperability rules.

Written by the indexing model from the issue text.