OTel export: event.name attribute is overwritten by the tracing call-site name
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 125k
- Forks
- 19.4k
- PR merge metrics
- PR metrics pending
Description
What version of Codex CLI is running?
0.152.1 locally; reproduced on every version we have telemetry for, 0.131.0 through 0.154.0. The behaviour is version-independent — see Additional information.
What subscription do you have?
Business / Enterprise
Which model were you using?
Not model-specific — this affects the telemetry export path, not inference.
What platform is your computer?
Darwin 25.6.0 arm64 arm
What terminal emulator and version are you using (if applicable)?
Not applicable — reproduces regardless of terminal.
Codex doctor report
not available
What issue are you seeing?
Every log record Codex exports over OTLP carries an event.name that is a Rust source location rather than the semantic event name. Instead of:
event.name = codex.sse_event
we receive:
event.name = event otel/src/events/session_telemetry.rs:863
This affects every event emitted through the session-telemetry macros — codex.api_request, codex.sse_event, codex.tool_result, codex.tool_decision, codex.user_prompt, codex.conversation_starts, codex.startup_phase, codex.turn_ttft, codex.turn_cost, codex.websocket_connect, codex.websocket_request, codex.websocket_event, codex.agent_communication, codex.sandbox_outcome, codex.auth_recovery, codex.network_proxy.policy_decision, codex.plugin_install_elicitation_sent, codex.plugin_install_suggestion.
No telemetry is lost — every event still arrives, with all its attributes. What is lost is the event's identity, so any consumer that groups or filters by event name gets nothing usable.
Root cause. Codex sets event.name as a tracing field, which becomes an OTLP attribute. Separately, the tracing → OpenTelemetry bridge populates the LogRecord's EventName field from the tracing metadata name — and because the emitting macros pass no name:, that metadata name is the crate's synthesised default, event <file>:<line>:
// codex-rs/otel/src/events/shared.rs:14
macro_rules! log_event {
($self:expr, $($fields:tt)*) => {{
tracing::event!(
target: $crate::targets::OTEL_LOG_ONLY_TARGET,
tracing::Level::INFO, // <-- no `name:` argument
$($fields)*
trace_event! (line 34) and log_and_trace_event! (line 52) have the same shape.
So each record ships two things called event.name: a correct attribute and a source-location field. Consumers that prefer the spec'd EventName field over a same-named attribute surface the source location; consumers that only read attributes look fine. That is why this can go unnoticed for a long time and then appear to "break" without a Codex release.
What steps can reproduce the bug?
You do not need any particular backend. Emit any session-telemetry event with OTLP log export enabled and inspect the payload:
LogRecord.event_name→event otel/src/events/session_telemetry.rs:<line>LogRecord.attributes["event.name"]→codex.sse_event
Any consumer that surfaces the former, or maps it onto an attribute of the same key, reports the source location.
To reproduce the downstream symptom end to end, export through an OpenTelemetry Collector on collector-contrib ≥ v0.154.0 using the googlecloud exporter, then query on event name. That release bumped the exporter's dependency on opentelemetry-operations-go/exporter/collector to v0.57.0, which added:
// exporter/collector/logs.go
// Add OTLP event_name field as "event.name" label
if eventName := logRecord.EventName(); eventName != "" {
if entry.Labels == nil {
entry.Labels = make(map[string]string)
}
entry.Labels["event.name"] = eventName
}
It runs after attributes are merged and assigns unconditionally, so a non-empty EventName always wins. The block is absent in exporter/collector v0.56.0 and present in v0.57.0; collector-contrib v0.153.0 and earlier export the semantic name correctly.
That version boundary bounds who is affected: any Codex user exporting to Cloud Logging on contrib ≥ v0.154.0 sees call-site names; anyone on ≤ v0.153.0 does not. Note that the guard is eventName != "" — producers that leave the LogRecord event name unset are unaffected entirely, which is why this is specific to the Rust tracing bridge rather than a property of the exporter.
What is the expected behavior?
event.name should be the semantic event name the code sets — codex.sse_event, codex.user_prompt, and so on — regardless of whether a consumer reads the LogRecord EventName field or the attribute of the same key. The two should not disagree.
Additional information
The substituted name is not stable across releases
This is the part that makes it more than a rename. The replacement is a source location, and source locations move, so the same line number means different events in different versions. Resolved against your release tags:
| Site | Resolves to |
|---|---|
otel/src/events/session_telemetry.rs:999 |
codex.tool_decision in 0.145.0-alpha.7…alpha.13; codex.sse_event in 0.150.0–0.153.2 |
otel/src/events/session_telemetry.rs:236 |
codex.startup_phase from 0.145.0 on; codex.turn_ttft in 0.143.0-alpha.32…0.145.0-alpha.2 |
otel/src/events/session_telemetry.rs:1012 |
codex.sse_event in 0.150.0–0.153.2; codex.sandbox_outcome in 0.138.0-alpha.6, 0.141.0-alpha.3, 0.142.0-alpha.4/6 |
otel/src/events/session_telemetry.rs:932 |
codex.sse_event in 0.133.0.x and 0.150.0-alpha.8; codex.user_prompt in 0.142.x and 0.143.0-alpha.4 |
otel/src/events/session_telemetry.rs:906 |
codex.user_prompt in 0.131.0-alpha.9; codex.sse_event in 0.148.0-alpha.21–0.149.1 |
So an affected consumer cannot recover the event type from a static lookup; it needs the client version as well. We resolved ours by reading the emitting file at the matching rust-v<version> tag for each (version, line) pair observed — 173 call sites across 100 versions, of which 10 are version-dependent as above. It works, but it is not something a telemetry consumer should need to do.
Emitting sites
| File | Emission style |
|---|---|
codex-rs/otel/src/events/session_telemetry.rs |
log_event! / trace_event! / log_and_trace_event! wrappers |
codex-rs/otel/src/tool_result.rs (~line 54) |
log_and_trace_event! |
codex-rs/model-provider/src/models_endpoint.rs |
tracing::event! directly |
codex-rs/core/src/agent_communication.rs |
tracing::info! with an event.name field |
codex-rs/network-proxy/src/network_policy.rs |
tracing::event!, event.name = POLICY_DECISION_EVENT_NAME |
Worth noting that agent_communication.rs uses tracing::info! rather than event!, so a change scoped only to the shared.rs wrappers would miss it.
Diagnostic logging (otel/src/metrics/client.rs, otel/src/provider.rs, and similar tracing::warn! / debug! calls) carries no event.name at all. Those seem correct as-is.
Observations on possible approaches
Entirely your call — noting what we considered, in case it is useful:
- Pass
name:totracing::event!so the metadata name is the semantic name. The collision then disappears rather than being arbitrated by whichever consumer is downstream.tracingrequiresname:to be a string literal at the call site, so theshared.rswrappers would need the name threaded through as a macro parameter — roughly 170 call sites, plus thetracing::info!site above. - Set the LogRecord event name at the appender/exporter layer from the existing
event.nameattribute. Narrower, same outcome. - Document the collision. Defensible if attribute-precedence consumers are the only supported target, though the collision is with a field in the OTel logs data model, so we would expect it to resurface as consumers modernise.
Related
#30936 (timeUnixNano=0) is a different defect but in the same bridge, and its analysis notes that opentelemetry-appender-tracing's on_event "builds the LogRecord (target, event name, severity, attributes)" — which is the step that fills EventName with the call site here. Both issues come down to what that layer does and does not populate, so a fix in this area may want to consider them together.
Secondary observation
The instrumentation scope on these records appears to be derived from the tracing target — e.g. codex_otel.log_only — and carries no scope version. Declaring a named, versioned instrumentation scope would make Codex telemetry easier to attribute and version on the consumer side. Unrelated to the bug above, but nearby.
What we are doing
A transform on our otel collector gateways as a mitigation:
transform/repair_codex_event_name:
error_mode: propagate
log_statements:
- context: log
statements:
- set(event_name, attributes["event.name"]) where attributes["event.name"] != nil and IsMatch(event_name, "^event .*:[0-9]+$")
We have a consumer-side workaround and are not blocked. We are reporting this because the underlying collision is client-side and version-independent, so other deployments will hit it as they upgrade collectors past v0.154.0. Happy to share the resolved site-to-name mapping — derived entirely from public source and release tags — if it is useful as a test fixture.
FWIW, no other harness' telemetry are affected by this.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in codex-rs/otel/src/events/shared.rs and inspect the log_event!, trace_event!, and log_and_trace_event! wrappers, then review the direct emitters in session_telemetry.rs, tool_result.rs, model-provider/src/models_endpoint.rs, core/src/agent_communication.rs, and network-proxy/src/network_policy.rs. Reproduce an OTLP log export and verify that LogRecord.event_name and the event.name attribute agree on the semantic Codex event name across these emission paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100