github / github/gh-aw

Feature request: fan out the agent engine's (Claude Code) OTLP traces to all GH_AW_OTLP_ENDPOINTS on the engine's behalf

Open
#38,915 1 comment 0 reactions 0 assignees View on GitHub
ai-generated ai-inspected community
Dominant language
Go
Stars
5.1k
Forks
541
Avg merge
5h 46m
Merged PRs (30d)
760

Description

🤖 _This issue has been generated by Claude Code._

## Feature request

Fan out the **agent engine's** (Claude Code) OTLP traces to **all** configured backends in `GH_AW_OTLP_ENDPOINTS`, performed by gh-aw on the engine's behalf — because the engine itself only supports a single OTLP endpoint and is unlikely to gain native multi-endpoint fan-out.

## Background

Claude Code's trace exporter is configured through the **standard OpenTelemetry SDK environment variables** ([Monitoring → Traces (beta)](https://code.claude.com/docs/en/monitoring-usage#traces-beta)): `OTEL_TRACES_EXPORTER`, `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, etc. These describe a **single** OTLP destination. The engine has no concept of a multi-endpoint fan-out set, and that's not something a general-purpose agent is likely to implement.

gh-aw, on the other hand, already supports multiple OTLP backends via `GH_AW_OTLP_ENDPOINTS` (the ordered fan-out set, per the OTEL observability spec §6). Today that fan-out is performed by gh-aw's own helper/runtime layer for the `gh-aw.*` lifecycle spans.

## Current behavior / gap

When a user configures N OTLP backends in `GH_AW_OTLP_ENDPOINTS`:

- gh-aw's lifecycle spans fan out to **all N** backends (helper/runtime layer).
- The engine can only target the **single primary endpoint** (`OTEL_EXPORTER_OTLP_ENDPOINT`, index 0), because that's all the standard OTel SDK env vars can express.

So the engine's spans (and, separately, the MCP gateway's spans — see #38901) reach only the primary backend; every non-primary backend receives an incomplete trace. This is the same primary-only limitation already reported for the gateway in #38901, but for the engine.

## Requested behavior

Have gh-aw provide a **local fan-out path** that the engine exports to, and re-export to every endpoint in `GH_AW_OTLP_ENDPOINTS` on the engine's behalf. For example:

- Point the engine's `OTEL_EXPORTER_OTLP_(TRACES_)ENDPOINT` at a **gh-aw-managed local OTLP receiver/forwarder** (sidecar/collector) running for the agent job.
- That forwarder fans the received engine spans out to all normalized endpoints following the **§6.6 fan-out operations flow** already specified for gh-aw: ordered endpoints, per-endpoint status (`success`/`transient_failure`/`permanent_failure`), partial-failure tolerance, and local-mirror persistence.

This reuses gh-aw's existing fan-out semantics rather than asking the engine to implement them.

## Why it's valuable

- **Backend parity**: every configured backend receives the engine's traces, not just the primary one.
- **Complete traces everywhere**: together with gateway fan-out (#38901) and W3C context propagation (#38905), all backends would see the full agent-job trace (engine `claude_code.*` spans + gateway `gateway.request`/`mcp.tool_call` spans nested under `gh-aw.agent.*`), instead of a complete picture only on the primary backend.
- **Engine-agnostic**: any engine that uses the standard single-endpoint OTel SDK configuration benefits from gh-aw doing the fan-out, with no per-engine work.

## Related

- #38901 — same primary-only limitation for the MCP gateway (`gh-aw-mcpg`) spans.
- #38905 — propagating W3C `TRACEPARENT`/`TRACESTATE` into the engine (so its spans join the trace in the first place); fan-out is orthogonal but complementary.
- OTEL observability spec §6 / §6.6 — the existing multi-endpoint fan-out operations flow this would extend to engine spans.

---

*Note: observed in a private repository. Repo/workflow names, run IDs, endpoint URLs, and trace/span IDs are redacted; the Claude Code documentation reference is to the public traces-beta page.*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.