Thread a correlation ID through routing, context builds, firewall events, and gateway calls

Summary

Introduce an optional correlation_id (or session_id/request_id pair) accepted by Router.route, ContextManager.build/_sync, the firewall facade, and gateway meta-tools, and stamped onto RouteTrace, BuildStats, FirewallStats, RoutingDecision.metadata, and log records — so one agent turn can be traced across both engines and the gateway.

Why this matters

Diagnosing "why did the agent see this prompt?" today requires manually matching separate artifacts: a RouteTrace, a BuildStats, firewall events, and gateway logs have no shared key. Every serious adopter rebuilds this correlation by hand. A first-class ID makes the existing rich diagnostics composable and is the prerequisite for the visualization/telemetry work to tell coherent per-turn stories.

Current evidence

• routing/trace.py (RouteTrace), envelope.py (BuildStats, FirewallStats, RoutingDecision) — none carries a caller-supplied correlation key (RoutingDecision has free-form metadata only).
• extras/otel.py creates spans but cannot link a route call to the subsequent build without caller plumbing.
• Open #378 (gateway telemetry export), #442 (HTML visualize), #459 (lifecycle hooks/BuildStats accounting), #370 (gateway diagnostics) all benefit; none introduces the shared key itself — this is the enabling primitive; reference them.

External context

W3C Trace Context / OpenTelemetry trace-span correlation is the ecosystem norm; GenAI semconv attributes assume a shared trace identity across steps.

Proposed implementation

1. Add correlation_id: str | None = None keyword to Router.route, ContextManager.build/build_sync/ingest_*, compact_tool_result, and ProxyRuntime dispatch (propagated from MCP request metadata where available).
2. Stamp into RouteTrace.extra, BuildStats, FirewallStats, RoutingDecision.metadata["contextweaver"], and logging records (via a LoggerAdapter or explicit fields).
3. When the [otel] extra is active, map to span attributes / links in OTelEventHook.
4. Default None keeps all outputs byte-identical.

AI-agent execution notes

• Inspect first: routing/trace.py, envelope.py, context/build.py, context/firewall_api.py, adapters/proxy_runtime.py, extras/otel.py, metrics.py.
• Schema impact: BuildStats/FirewallStats/RouteTrace field additions require make schemas and the drift gate.
• Preserve determinism: the ID is caller-supplied, never auto-generated in core paths (no uuid4 in pipelines — to_routing_decision already mints IDs only at the envelope edge).

Acceptance criteria

• One ID passed at route() and build_sync() appears in the trace, stats, firewall events, and (when enabled) OTel attributes for that turn.
• None default: all existing fixtures/golden tests pass byte-identical.
• Schemas regenerated; serde round-trips with and without the field.

Test plan

Unit tests per surface; an end-to-end test running route → ingest → build asserting the same ID on all artifacts; OTel hook test (existing tests/test_otel.py pattern); make ci.

Documentation plan

docs/integration_otel.md and troubleshooting "tracing a turn" section, CHANGELOG, schema docs.

Migration and compatibility notes

Additive optional field on serialized payloads; older payloads load via default-tolerant from_dict. Not expected to require migration.

Risks and tradeoffs

Touches many signatures (keyword-only mitigates); risk of partial propagation (audit with the e2e test); alternative contextvars-based implicit propagation rejected for explicitness and sync/async boundary simplicity.

Suggested labels

architecture, developer-experience, reliability

Start by mapping the call paths in routing/trace.py, envelope.py, context/build.py, context/firewall_api.py, adapters/proxy_runtime.py, extras/otel.py, and metrics.py. Review the existing tests/test_otel.py pattern, then add surface-level and route-to-build tests covering shared IDs, None defaults, schema round-trips, and OTel output. Done means make schemas and make ci pass with existing fixtures byte-identical.