openai / openai/codex

OTel export: event.name attribute is overwritten by the tracing call-site name

Open
#42,837 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug CLI
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

What version of Codex CLI is running?

0.152.1 locally; reproduced on every version we have telemetry for, 0.131.0 through 0.154.0. The behaviour is version-independent — see Additional information.

What subscription do you have?

Business / Enterprise

Which model were you using?

Not model-specific — this affects the telemetry export path, not inference.

What platform is your computer?

Darwin 25.6.0 arm64 arm

What terminal emulator and version are you using (if applicable)?

Not applicable — reproduces regardless of terminal.

Codex doctor report
not available
What issue are you seeing?

Every log record Codex exports over OTLP carries an event.name that is a Rust source location rather than the semantic event name. Instead of:

event.name = codex.sse_event

we receive:

event.name = event otel/src/events/session_telemetry.rs:863

This affects every event emitted through the session-telemetry macros — codex.api_request, codex.sse_event, codex.tool_result, codex.tool_decision, codex.user_prompt, codex.conversation_starts, codex.startup_phase, codex.turn_ttft, codex.turn_cost, codex.websocket_connect, codex.websocket_request, codex.websocket_event, codex.agent_communication, codex.sandbox_outcome, codex.auth_recovery, codex.network_proxy.policy_decision, codex.plugin_install_elicitation_sent, codex.plugin_install_suggestion.

No telemetry is lost — every event still arrives, with all its attributes. What is lost is the event's identity, so any consumer that groups or filters by event name gets nothing usable.

Root cause. Codex sets event.name as a tracing field, which becomes an OTLP attribute. Separately, the tracing → OpenTelemetry bridge populates the LogRecord's EventName field from the tracing metadata name — and because the emitting macros pass no name:, that metadata name is the crate's synthesised default, event <file>:<line>:

// codex-rs/otel/src/events/shared.rs:14
macro_rules! log_event {
    ($self:expr, $($fields:tt)*) => {{
        tracing::event!(
            target: $crate::targets::OTEL_LOG_ONLY_TARGET,
            tracing::Level::INFO,          // <-- no `name:` argument
            $($fields)*

trace_event! (line 34) and log_and_trace_event! (line 52) have the same shape.

So each record ships two things called event.name: a correct attribute and a source-location field. Consumers that prefer the spec'd EventName field over a same-named attribute surface the source location; consumers that only read attributes look fine. That is why this can go unnoticed for a long time and then appear to "break" without a Codex release.

What steps can reproduce the bug?

You do not need any particular backend. Emit any session-telemetry event with OTLP log export enabled and inspect the payload:

  • LogRecord.event_nameevent otel/src/events/session_telemetry.rs:<line>
  • LogRecord.attributes["event.name"]codex.sse_event

Any consumer that surfaces the former, or maps it onto an attribute of the same key, reports the source location.

To reproduce the downstream symptom end to end, export through an OpenTelemetry Collector on collector-contrib ≥ v0.154.0 using the googlecloud exporter, then query on event name. That release bumped the exporter's dependency on opentelemetry-operations-go/exporter/collector to v0.57.0, which added:

// exporter/collector/logs.go
// Add OTLP event_name field as "event.name" label
if eventName := logRecord.EventName(); eventName != "" {
  if entry.Labels == nil {
    entry.Labels = make(map[string]string)
  }
  entry.Labels["event.name"] = eventName
}

It runs after attributes are merged and assigns unconditionally, so a non-empty EventName always wins. The block is absent in exporter/collector v0.56.0 and present in v0.57.0; collector-contrib v0.153.0 and earlier export the semantic name correctly.

That version boundary bounds who is affected: any Codex user exporting to Cloud Logging on contrib ≥ v0.154.0 sees call-site names; anyone on ≤ v0.153.0 does not. Note that the guard is eventName != "" — producers that leave the LogRecord event name unset are unaffected entirely, which is why this is specific to the Rust tracing bridge rather than a property of the exporter.

What is the expected behavior?

event.name should be the semantic event name the code sets — codex.sse_event, codex.user_prompt, and so on — regardless of whether a consumer reads the LogRecord EventName field or the attribute of the same key. The two should not disagree.

Additional information
The substituted name is not stable across releases

This is the part that makes it more than a rename. The replacement is a source location, and source locations move, so the same line number means different events in different versions. Resolved against your release tags:

Site Resolves to
otel/src/events/session_telemetry.rs:999 codex.tool_decision in 0.145.0-alpha.7alpha.13; codex.sse_event in 0.150.00.153.2
otel/src/events/session_telemetry.rs:236 codex.startup_phase from 0.145.0 on; codex.turn_ttft in 0.143.0-alpha.320.145.0-alpha.2
otel/src/events/session_telemetry.rs:1012 codex.sse_event in 0.150.00.153.2; codex.sandbox_outcome in 0.138.0-alpha.6, 0.141.0-alpha.3, 0.142.0-alpha.4/6
otel/src/events/session_telemetry.rs:932 codex.sse_event in 0.133.0.x and 0.150.0-alpha.8; codex.user_prompt in 0.142.x and 0.143.0-alpha.4
otel/src/events/session_telemetry.rs:906 codex.user_prompt in 0.131.0-alpha.9; codex.sse_event in 0.148.0-alpha.210.149.1

So an affected consumer cannot recover the event type from a static lookup; it needs the client version as well. We resolved ours by reading the emitting file at the matching rust-v<version> tag for each (version, line) pair observed — 173 call sites across 100 versions, of which 10 are version-dependent as above. It works, but it is not something a telemetry consumer should need to do.

Emitting sites
File Emission style
codex-rs/otel/src/events/session_telemetry.rs log_event! / trace_event! / log_and_trace_event! wrappers
codex-rs/otel/src/tool_result.rs (~line 54) log_and_trace_event!
codex-rs/model-provider/src/models_endpoint.rs tracing::event! directly
codex-rs/core/src/agent_communication.rs tracing::info! with an event.name field
codex-rs/network-proxy/src/network_policy.rs tracing::event!, event.name = POLICY_DECISION_EVENT_NAME

Worth noting that agent_communication.rs uses tracing::info! rather than event!, so a change scoped only to the shared.rs wrappers would miss it.

Diagnostic logging (otel/src/metrics/client.rs, otel/src/provider.rs, and similar tracing::warn! / debug! calls) carries no event.name at all. Those seem correct as-is.

Observations on possible approaches

Entirely your call — noting what we considered, in case it is useful:

  1. Pass name: to tracing::event! so the metadata name is the semantic name. The collision then disappears rather than being arbitrated by whichever consumer is downstream. tracing requires name: to be a string literal at the call site, so the shared.rs wrappers would need the name threaded through as a macro parameter — roughly 170 call sites, plus the tracing::info! site above.
  2. Set the LogRecord event name at the appender/exporter layer from the existing event.name attribute. Narrower, same outcome.
  3. Document the collision. Defensible if attribute-precedence consumers are the only supported target, though the collision is with a field in the OTel logs data model, so we would expect it to resurface as consumers modernise.
Related

#30936 (timeUnixNano=0) is a different defect but in the same bridge, and its analysis notes that opentelemetry-appender-tracing's on_event "builds the LogRecord (target, event name, severity, attributes)" — which is the step that fills EventName with the call site here. Both issues come down to what that layer does and does not populate, so a fix in this area may want to consider them together.

Secondary observation

The instrumentation scope on these records appears to be derived from the tracing target — e.g. codex_otel.log_only — and carries no scope version. Declaring a named, versioned instrumentation scope would make Codex telemetry easier to attribute and version on the consumer side. Unrelated to the bug above, but nearby.

What we are doing

A transform on our otel collector gateways as a mitigation:

transform/repair_codex_event_name:
    error_mode: propagate
    log_statements:
      - context: log
        statements:
          - set(event_name, attributes["event.name"]) where attributes["event.name"] != nil and IsMatch(event_name, "^event .*:[0-9]+$")

We have a consumer-side workaround and are not blocked. We are reporting this because the underlying collision is client-side and version-independent, so other deployments will hit it as they upgrade collectors past v0.154.0. Happy to share the resolved site-to-name mapping — derived entirely from public source and release tags — if it is useful as a test fixture.

FWIW, no other harness' telemetry are affected by this.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in codex-rs/otel/src/events/shared.rs and inspect the log_event!, trace_event!, and log_and_trace_event! wrappers, then review the direct emitters in session_telemetry.rs, tool_result.rs, model-provider/src/models_endpoint.rs, core/src/agent_communication.rs, and network-proxy/src/network_policy.rs. Reproduce an OTLP log export and verify that LogRecord.event_name and the event.name attribute agree on the semantic Codex event name across these emission paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.