[📑 Docs]: Epic: Agent-First Documentation (Discovery Layer & Docs MCP)

### What Dev Docs changes are you proposing?

## What Dev Docs changes are you proposing?

Following the docs working group session with Lukasz, the direction from the previous [epic](https://github.com/orgs/asyncapi/discussions/3532) sharpened a lot.

- The work splits into **two distinct streams** that were previously tangled together:
- Making our docs discoverable by *any* agent, and
- Building *our own* MCP for the docs.
- We are **not building vectorization as a separate effort**. Chunking → embedding → indexing is handled end-to-end by [open crane](https://github.com/derberg/asyncapi-knowledge-mcp/tree/main/.opencrane), which ships an embedded Milvus DB inside the MCP. So "do we need a vector database" is no longer an open question; the tool carries it.
- The old **upfront "Semantic Document Restructuring" audit is out**. You can't fix everything by editing content ahead of time; the chunking strategy carries most of it, and the rest becomes **iterative follow-ups** the working group triages after each build.
- **Hosting and the website chatbot are deferred.** An MCP can run locally as a library talking to a local agent over stdin; no hosting needed to get value. DigitalOcean and a `.com` chatbot come later.

## Why

As of 2026, a large share of "contributors" are humans working through agents, or agents acting on their behalf (Cursor, Claude Code, Copilot, LLM crawlers).

Our docs are optimized for visual consumption, which is high token-noise and error-prone for retrieval.

Giving agents a clean discovery layer and a queryable MCP reduces onboarding friction and keeps AI-generated AsyncAPI specs closer to the current standard.

## Stream A: Machine-readable discovery (start here)

Independent of our MCP. This makes our docs consumable by *anyone's* agent, and it's the faster, easier win. There's already a [website issue](https://github.com/asyncapi/website/issues/4408) for the core of this, so pick it up rather than opening a new one.

- [ ] Expose `/llms.txt` on asyncapi.com: A navigation map of our docs for LLMs *(existing website issue)*
- [ ] Expose `/llms-full.txt`: An essential community + spec docs concatenated into a single high-context Markdown file
- [ ] **Investigate:** A clean per-page Markdown (a `readmd`-style raw `.md` view of each page, no HTML). This is an emerging pattern, not as standardized as `llms.txt`, so decide whether it's worth doing now.

**Owner:** @princerajpoot20

## Stream B: AsyncAPI Docs MCP (via open crane)

Our own MCP server so agents can semantically query our docs. Bigger, longer effort. Best developed with Claude Code.

### Prerequisites

- [ ] Create an official AsyncAPI **PyPI account** so we can publish the MCP package *@thulieblack*
- [ ] **Spike:** Get hands-on with open crane. Clone the AsyncAPI knowledge MCP repo (and Lukasz's [open source Europe repo](https://github.com/opensourceeurope/ose-knowledge-mcp), which tracks the latest open crane version), install open crane locally, run `fetch` / `llms` / `chunk`, and examine what each command produces.

### Build pipeline (open crane handles this end-to-end)

- [ ] **Config:** Define doc sources. Our content is already consolidated into the website (generator, community, JSON schema all push there), so sources are in good shape
- [ ] **Fetch:** Clone repos and pull Markdown into the repo
- [ ] **Generate llms.txt** from the fetched content
- [ ] **Chunking:** The single most critical step. Strategic decisions on tables, long lists, and heading structure live here
- [ ] **Embedding:** Generate the vectors that make semantic search work
- [ ] **Indexing** into an embedded **Milvus DB** (lightweight, Linux Foundation, bundled inside the MCP, so no heavy DB setup)
- [ ] **Build** the MCP server
- [ ] **Publish** the MCP package to PyPI
- [ ] **Wire open crane into the website repo CI**, including the manual step to copy generated `llms` files into the correct public location

## Stream C: Agent skills

MCP on its own gives agents semantic access to our content, but *no instructions on how to use it*. Skills close that gap and make retrieval meaningfully better.

- [ ] Author skills that tell agents how to search and use the AsyncAPI docs MCP effectively
- [ ] Account for the cross-tool reality: Skills are currently built for Claude Code, while Cursor and others have their own skill/marketplace systems, so decide how far we support each

## Stream D: Iterative content follow-ups (replaces the old restructuring audit)

Not a one-time audit. After each MCP build, the working group reviews chunk results and fixes what surfaces.

- [ ] Working group reviews chunk output after each build
- [ ] Fix issues chunking exposes: Add intro prose before tables, split oversized tables / long lists, tighten heading hierarchy where it breaks chunking

Expect a steady stream of small doc PRs from this.

## Deferred (not now)

- Website chatbot on asyncapi.com
- DigitalOcean hosting (DO is renewed through next year, but not needed to get the MCP working)

## Out of scope

- Using AI to *write* the docs: We're making existing docs readable for AI, not generating them
- Website redesign: This is structural/metadata, not a UI overhaul

## Measuring success

- **Zero-hallucination check:** Qualitative agent testing (e.g. "explain the AsyncAPI TSC voting process") before vs. after the discovery layer + MCP land
- **Token efficiency:** Fewer tokens needed for an agent to accurately summarize a contribution guide

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

### Are you willing to work on this issue?

Yes I am willing to submit a PR!

Treat this as an epic rather than a single starter task. Begin with the linked website issue 4408 for the discovery layer, then clone the AsyncAPI knowledge MCP and open source Europe repositories and run open crane's fetch, llms, and chunk commands. A contribution is complete when it targets one unchecked stream item and demonstrates the output or decision described there.

Written by the indexing model from the issue text.