API docs: move all documentation controls out of protos into 1:1 sidecar RST files
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 437
Description
*Title*: Move all API documentation controls out of protos into 1:1 sidecar RST files
*Description*:
>Today the autogenerated API docs (`docs/tools/protodoc/`) derive their RST prose from comment blocks embedded directly in the `.proto` files. The `RstFormatVisitor` walks the message/enum/field tree and, at each node, inlines the leading comment (via `annotations.without_annotations(comment.raw)`) as RST, interleaved with auto-generated structure (field type tables, JSON examples, cross-ref anchors, `:ref:` links).
>
>This is fragile for API editors, who must simultaneously know:
>
>- the API/proto semantics,
>- RST syntax,
>- and the comment-embedding + annotation conventions (which are hard to get right without trial-and-error, and are effectively tribal knowledge).
>
>On top of that, you cannot edit the RST in a native editor — the prose lives inside proto comments, so multi-context RST editing requires editor-specific back-flips that work badly if at all.
>
>**Proposal (design "D+A"): separate structure from prose, and give the sidecar complete control.**
>
>- **Protos own structure only (requirement).** Proto comments retain *only* structural/semantic annotations that are genuine facts about the API — e.g. `[#next-free-field]`, extension category/security-posture metadata, `[#not-implemented-hide:]`, WIP status. **All documentation controls live in the sidecar; protos carry none by requirement.**
>- **A 1:1 sidecar RST file per proto** (`foo.proto` ↔ `foo.proto.rst`) owns 100% of the prose.
>- **Symbolic (FQN) addressing.** Each RST block is keyed by the fully-qualified name protodoc already computes (`normalize_type_context_name(ctx.name)`), e.g. `envoy.config.cluster.v3.Cluster.http2_protocol_options`. This is the same key space `protodoc_manifest.yaml` already uses, so no new addressing scheme is invented. FQN keys are robust to field reordering, and for a stable versioned API (`v3`) renames are breaking changes anyway.
>- **Replace-only overlay, never merge.** A node has exactly one prose source: the sidecar (or explicitly nothing). Prose is *never* merged from proto comments, avoiding the "which source wins / why is my doc blank" ambiguity.
>- **Doc opt-out lives in the sidecar** (e.g. a `:no-doc:` marker), kept distinct from proto `[#not-implemented-hide:]`. Proto never encodes doc-policy.
>
>**Mechanics to implement:**
>
>1. **Overlay handling (visitor change).** Repoint the single choke point (`_comment()` / `_header_from_file()` in `docs/tools/protodoc/protodoc.py`) to look up prose by FQN in a loaded sidecar. Annotations still come from `comment.annotations`; only prose stops being read from `comment.raw`. Small, local change; can run in parallel with the old behaviour during migration.
>2. **Linting (the real safety net).** Because the visitor walks every node, we get an authoritative set of documentable FQNs. Lint compares it against the sidecar keys with four hard-error classes:
> - orphan sidecar key (FQN no proto node produces — typo/rename/removal),
> - undocumented node (no sidecar prose and no explicit `:no-doc:`),
> - prose-in-proto leak (`without_annotations(raw)` non-empty — enforces the "protos own no docs" requirement),
> - structural-annotation-in-sidecar leak.
>3. **Migration.** One-shot mechanical extraction: (a) dump pass writes `FQN → without_annotations(comment.raw)` into sidecars; (b) strip pass rewrites proto comments down to annotations only; (c) equivalence check re-generates docs from sidecars and diffs against pre-migration output (expect byte-identical modulo whitespace) as the acceptance test.
>4. **Upstream-drift handling.** We are *not* retaining/reusing upstream proto prose long-term — relying on upstream is the lateral coupling this resolves. For generic/upstream-owned nodes (auto-scoped via the existing `_is_envoy_proto` / `envoy_prefixes` distinction), store a cheap fingerprint (hash of the upstream comment text) alongside sidecar prose, and add a periodic drift-detection lint that flags "upstream description changed since last sync" for human review. This preserves replace-only (we never render upstream text) while making drift a reviewable event rather than a silent obligation, and degrades gracefully if upstream's comment format changes in future.
>
>**Reused machinery:** FQN keying, annotation extraction, envoy-prefix classification, and the visitor's total node walk all already exist. Net-new code is a sidecar loader, the four-class lint, and the migration dump/strip/diff script.
[optional *Relevant Links*:]
>- `docs/tools/protodoc/protodoc.py` — `RstFormatVisitor`, `_comment()`, `_header_from_file()`
>- `tools/api_proto_plugin/annotations.py` — `extract_annotations` / `without_annotations`
>- `tools/api_proto_plugin/type_context.py` — `Comment`, `file_level_comments`
>- `docs/tools/protodoc/protodoc_manifest.yaml` — existing FQN-keyed field metadata (precedent for symbolic addressing)
Contributor guide
Research direction
Start with RstFormatVisitor, _comment(), and _header_from_file() in docs/tools/protodoc/protodoc.py, then review annotations.py and type_context.py for comment and annotation handling. Use protodoc_manifest.yaml as the FQN-keying precedent; done means sidecar prose replaces proto prose, linting catches the four specified error classes, and migration preserves generated documentation apart from permitted whitespace differences.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100