openai / openai/codex

OTel: skill invocations are only observable as a metric counter — emit a log event (`codex.skill.injected`) for fleet attribution

Open
#45,357 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

CLI enhancement skills
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

What variant of Codex are you using?

CLI (the instrumentation lives in codex-rs core, so App / app-server are equally affected)

What feature would you like to see?

Emit skill invocations as an OTel log event (codex.skill.injected), alongside the existing metric counter of the same name — the same pattern codex.tool_decision / codex.tool_result already follow.

Use case

We run Codex across an enterprise fleet (thousands of managed devices) and collect OTel telemetry centrally to measure skill adoption and per-team ROI. Today skill usage is observable only as the codex.skill.injected counter, and a counter is structurally unable to support this, for two independent reasons:

  1. The metrics resource carries no identity, by design. The metrics client builds its own resource with only service.*, env and OS attributes (codex-rs/otel/src/metrics/client.rs, os_resource_attributes), while host.name is attached only for ResourceKind::Logs (codex-rs/otel/src/provider.rs, resource_attributes). We verified this in production on 0.148.0: skill-counter datapoints arrive with env / os / os_version / service.* only, while log events from the very same process do carry host.name. The code is unchanged on main as of 3abbf9f (2026-09-14). Counter datapoints therefore cannot be attributed to a device or user, even server-side.

  2. No conversation linkage. The counter has no conversation/turn identity, so skill usage cannot be joined to codex.api_request / token-usage events to measure per-skill cost. The log_event! common fields (conversation.id, originator, app.version, terminal.type, model, slug, auth mode) provide exactly this.

Root-cause analysis and a potential approach

The insertion point already exists: emit_explicit_skill_invocations and maybe_emit_implicit_skill_invocation (codex-rs/core/src/skills.rs) fan each invocation out to the metric counter and to analytics_events_client.track_skill_invocations — i.e. OpenAI's internal analytics already receives exactly this event; OTel consumers are the only ones who cannot.

A minimal change (~50 lines + a test), validated against main @ 3abbf9f (rust 1.95.0):

  • codex-rs/otel/src/events/session_telemetry.rs: add skill_injected(&self, skill_name, status, invoke_type, reasoning_effort, plugin_id: Option<&str>) using log_event! (log-only target, mirroring tool_decision). It carries the skill name and the same invoke_type / reasoning_effort / plugin_id dimensions the counter has since #40724; model / slug come from the macro's common fields.
  • codex-rs/core/src/skills.rs: call it right next to the existing counter emission in both the explicit and implicit paths, so the implicit path's per-turn dedup is shared.

Validation: cargo check -p codex-otel -p codex-core clean, the new traced_test-based test skill_injected_records_log_event passes, cargo fmt --check clean.

Validated diff (115 insertions, 3 files, against main @ 3abbf9f)
diff --git a/codex-rs/core/src/skills.rs b/codex-rs/core/src/skills.rs
index ef610de3..01b1b8a3 100644
--- a/codex-rs/core/src/skills.rs
+++ b/codex-rs/core/src/skills.rs
@@ -73,6 +73,13 @@ pub(crate) async fn emit_explicit_skill_invocations(
                 ("reasoning_effort", reasoning_effort.as_str()),
             ],
         );
+        turn_context.session_telemetry.skill_injected(
+            skill.name.as_str(),
+            status,
+            "explicit",
+            reasoning_effort.as_str(),
+            skill.plugin_id.as_deref(),
+        );
     }
 
     let injected_host_skill_prompts = turn_context
@@ -195,6 +202,13 @@ pub(crate) async fn maybe_emit_implicit_skill_invocation(
             ("reasoning_effort", reasoning_effort.as_str()),
         ],
     );
+    turn_context.session_telemetry.skill_injected(
+        skill_name.as_str(),
+        "ok",
+        "implicit",
+        reasoning_effort.as_str(),
+        invocation.plugin_id.as_deref(),
+    );
     sess.services
         .analytics_events_client
         .track_skill_invocations(
diff --git a/codex-rs/core/tests/suite/otel.rs b/codex-rs/core/tests/suite/otel.rs
index 64b8790f..69cab09b 100644
--- a/codex-rs/core/tests/suite/otel.rs
+++ b/codex-rs/core/tests/suite/otel.rs
@@ -1199,6 +1199,70 @@ fn network_policy_decisions_omit_source_and_destination() {
     }
 }
 
+#[test]
+#[traced_test]
+fn skill_injected_records_log_event() {
+    let telemetry = SessionTelemetry::new(
+        ThreadId::new(),
+        "gpt-5.5",
+        "gpt-5.5",
+        /*account_id*/ None,
+        /*account_email*/ None,
+        Some(TelemetryAuthMode::ApiKey),
+        "Codex_Desktop".to_string(),
+        /*log_user_prompts*/ false,
+        "tty".to_string(),
+        SessionSource::Cli,
+    );
+
+    telemetry.skill_injected(
+        "linear", "ok", "explicit", /*reasoning_effort*/ "default", /*plugin_id*/ None,
+    );
+    telemetry.skill_injected(
+        "my-plugin:docs-review",
+        "ok",
+        "implicit",
+        /*reasoning_effort*/ "high",
+        /*plugin_id*/ Some("my-plugin"),
+    );
+
+    logs_assert(|lines: &[&str]| {
+        let bare = lines
+            .iter()
+            .find(|line| line.contains("codex.skill.injected") && line.contains("skill=linear"))
+            .ok_or_else(|| "missing skill injection event for bare skill".to_string())?;
+        if !bare.contains("status=ok")
+            || !bare.contains("invoke_type=explicit")
+            || !bare.contains("reasoning_effort=default")
+        {
+            return Err("bare skill event missing status/invoke_type/reasoning_effort".to_string());
+        }
+        if bare.contains("plugin_id=") {
+            return Err("bare skill event unexpectedly included a plugin id".to_string());
+        }
+
+        // The log event carries the skill name as-is, including a plugin
+        // namespace prefix.
+        let namespaced = lines
+            .iter()
+            .find(|line| {
+                line.contains("codex.skill.injected")
+                    && line.contains("skill=my-plugin:docs-review")
+            })
+            .ok_or_else(|| "missing skill injection event for plugin skill".to_string())?;
+        if !namespaced.contains("invoke_type=implicit")
+            || !namespaced.contains("reasoning_effort=high")
+        {
+            return Err("plugin skill event missing invoke_type/reasoning_effort".to_string());
+        }
+        if !namespaced.contains("plugin_id=my-plugin") {
+            return Err("plugin skill event missing plugin id".to_string());
+        }
+
+        Ok(())
+    });
+}
+
 #[test]
 #[traced_test]
 fn sandbox_outcome_event_records_outcome() {
diff --git a/codex-rs/otel/src/events/session_telemetry.rs b/codex-rs/otel/src/events/session_telemetry.rs
index 5e8f7ceb..39daa1b1 100644
--- a/codex-rs/otel/src/events/session_telemetry.rs
+++ b/codex-rs/otel/src/events/session_telemetry.rs
@@ -1124,6 +1124,43 @@ impl SessionTelemetry {
         }
     }
 
+    /// Records a skill injection as a log event, alongside the existing
+    /// `codex.skill.injected` metric counter emitted by the caller.
+    ///
+    /// The counter alone cannot support per-conversation or per-device
+    /// attribution: the metrics resource intentionally carries no host
+    /// identity and the counter has no conversation linkage. The log event
+    /// complements it with the log resource and the per-session metadata
+    /// (conversation id, originator) attached by `log_event!`.
+    pub fn skill_injected(
+        &self,
+        skill_name: &str,
+        status: &str,
+        invoke_type: &str,
+        reasoning_effort: &str,
+        plugin_id: Option<&str>,
+    ) {
+        match plugin_id {
+            Some(plugin_id) => log_event!(
+                self,
+                event.name = "codex.skill.injected",
+                skill = %skill_name,
+                status = %status,
+                invoke_type = %invoke_type,
+                reasoning_effort = %reasoning_effort,
+                plugin_id = %plugin_id,
+            ),
+            None => log_event!(
+                self,
+                event.name = "codex.skill.injected",
+                skill = %skill_name,
+                status = %status,
+                invoke_type = %invoke_type,
+                reasoning_effort = %reasoning_effort,
+            ),
+        }
+    }
+
     pub fn sandbox_outcome(
         &self,
         tool_name: &str,

Related issues

  • #11052 — asked for skill_name on tool events; closed NOT_PLANNED for lack of upvotes. This request is the complementary shape: a dedicated event also covers explicit $skill mentions, which create no tool call and are invisible to tool-event attribution.
  • #41760 — asks for a SkillInvocation event in the local log stream (logs_2.sqlite) via a tracing event on target codex_analytics::skill_invocation. That target is filtered out of OTLP export (is_log_export_target in codex-rs/otel/src/targets.rs only passes codex_otel*), so even if merged it would not reach an OTel collector. This request is the exporter-facing counterpart; going through log_event! also attaches conversation.id / originator / host.name.
  • #35650 / #40724 — extend the counter's attributes. #40724 landed plugin_id / model_slug / reasoning_effort; the remaining gaps (identity, conversation linkage) are what a counter cannot carry at all, which is why a log event is proposed rather than more tags.
  • #17132 / #39906 — hook events for skill invocations. Hooks are per-user, client-side configuration and cannot serve centrally managed fleet telemetry; the OTel exporter is the managed path.
  • #35869 — SkillsExtension-managed plugin skills emit no counter at all. We reproduced this on 0.148.0 (SKILL.md demonstrably injected per the session rollout, yet nothing emitted). A log event at the core fanout keeps parity for that future fix as well.
Additional information

For comparison, Claude Code exposes both a skill_activated log event and a skill.name attribute on API-request events; we use those for the equivalent measurements on that side of our fleet. Happy to share further environment details or test evidence.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with emit_explicit_skill_invocations and maybe_emit_implicit_skill_invocation in codex-rs/core/src/skills.rs, then inspect the existing telemetry methods and log_event! usage in codex-rs/otel/src/events/session_telemetry.rs. Add coverage beside the OTel tests in codex-rs/core/tests/suite/otel.rs and run the named cargo checks, the traced test, and cargo fmt --check; done means both invocation paths produce the documented event fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.