Emit attributable SkillInvocation events to the local Codex log stream
Open
Nobody has claimed this yet.
CLI
enhancement
skills
- Dominant language
- Rust
- Stars
- 125k
- Forks
- 19.4k
- PR merge metrics
- PR metrics pending
Description
Summary
Codex already detects explicit and implicit skill invocation and carries enough attribution to its remote analytics reducer, metric, and extension contributors. The ordinary local log stream does not expose an equivalent first-class event, so local audit tools cannot answer which skill was used for a particular thread/turn without inferring from prompt text or generic file reads.
This request is for a versioned, privacy-bounded local tracing event at the existing AnalyticsEventsClient::track_skill_invocations boundary. It is distinct from #39906's external hook notification and #35650's remote metric enrichment: the use case is queryable local runtime evidence in logs_2.sqlite, including when remote analytics is disabled and without enabling a write-capable hook handler.
Reproduction on 0.151.0
- Start Codex/app-server with a normal state database.
- Invoke a skill explicitly with a
Skillinput item, or implicitly by reading/running a skill resource. - Query new rows in
~/.codex/sqlite/logs_2.sqlitefor a skill-invocation target/name.
Actual behavior: there is no attributable local skill-use event. codex.skill.injected is an OpenTelemetry counter and skill_invocation is reduced for optional remote analytics; neither is a first-class row in the default local log stream.
Expected behavior: one local event for the SkillInvocation supplied to the shared analytics client, with enough fields to attribute use while excluding raw resource locators.
For resource-backed implicit invocations, cardinality should match the existing
remote reducer: one occurrence per (turn_id, resource_id). Repeated explicit
occurrences remain distinct.
Root cause
codex-rs/core/src/skills.rs and codex-rs/ext/skills/src/tools/mod.rs both converge on AnalyticsEventsClient::track_skill_invocations. That method currently enqueues CustomAnalyticsFact::SkillInvoked only when the analytics client is enabled. The state runtime already attaches codex_state::log_db::LogDbLayer in both app-server and TUI, and its default filter accepts an INFO tracing target, so no new database or hook system is required.
The extension path has one additional cardinality edge: repeated zero-cursor
reads of the same Skill main resource can call the analytics client twice in
one turn, while AnalyticsReducer intentionally collapses the second implicit
resource invocation. A local event emitted before that reducer must apply the
same rule earlier. The proposed patch reuses the existing active-turn Skill
telemetry lifecycle for that bounded set; it does not add client-global or
persistent dedupe state.
Caller census after the patch is four Rust files: the analytics implementation, the core production caller family, the extension production caller family, and the new direct test. There are two production caller families.
Proposed event schema
target=codex_analytics::skill_invocation
event_name=codex.skill_invocation
schema_version=1
thread_id=<thread>
turn_id=<turn>
skill_name=<name>
skill_resource_id=<one-way digest of host path or resource id>
skill_location=host|resource
skill_scope=user|repo|system|admin|unknown
plugin_id=<id|none>
remote_plugin_id=<id|none>
invocation_type=explicit|implicit
model=<slug>
product_client_id=<originator>
The event should be emitted before optional remote delivery so analytics.enabled=false does not erase local audit evidence. It should not include the raw host path, resource URI, supplied resource skill ID, prompt text, or skill body.
Two v1 scope notes are deliberate and should be explicit:
- For host skills, the proposed local resource identity hashes the raw absolute path. That is privacy-preserving relative to logging the path, but it is not joinable with the remote reducer's repo-normalized
skill_id, and a repo skill's local identity changes when the checkout moves. A later schema could normalize repo-scoped paths if cross-surface joining is desired. - The default SQLite persistence path is wired by app-server and TUI. Other runtimes such as headless exec can emit the tracing event to their configured subscriber but do not attach the
logs_2.sqlitelayer; this request does not claim SQLite coverage for every binary mode.
Two reproducible patches
The same five-file patch (277 additions, 6 deletions) was applied and tested independently against:
- Exact installed release
rust-v0.151.0(78c290807ce710180111df227df3b7a4fe845452): local tip23bd325ae06d056ba42a1b67970fd5c61cbb3096; patch SHA-256957762a6f00b3dbc65d51a8a657bb07e6938da5d86f7235c9d4c5dac7d47f72f. - Current upstream main (
b7cd519c767c8fd4bc3581d9bc92fbab37a768c1): local tip70f779e85befac1bbc4e20c8d6e6ee35c4c49700; patch SHA-256957762a6f00b3dbc65d51a8a657bb07e6938da5d86f7235c9d4c5dac7d47f72f.
The patch changes only:
codex-rs/analytics/src/client.rscodex-rs/core/tests/suite/skills.rscodex-rs/ext/skills/Cargo.tomlcodex-rs/ext/skills/src/telemetry.rscodex-rs/ext/skills/src/tools/mod.rs
Exact rust-v0.151.0 patch (SHA-256 957762a6…f72f)
diff --git a/codex-rs/analytics/src/client.rs b/codex-rs/analytics/src/client.rs
index 7b5056acc..2963a8692 100644
--- a/codex-rs/analytics/src/client.rs
+++ b/codex-rs/analytics/src/client.rs
@@ -18,6 +18,7 @@ use crate::facts::ExternalAgentConfigImportFailureInput;
use crate::facts::HookRunFact;
use crate::facts::HookRunInput;
use crate::facts::ImagePreparationFact;
+use crate::facts::InvocationType;
use crate::facts::PluginInstallFailedInput;
use crate::facts::PluginInstallRequested;
use crate::facts::PluginInstallRequestedInput;
@@ -26,6 +27,7 @@ use crate::facts::PluginMeasurementsInput;
use crate::facts::PluginState;
use crate::facts::PluginStateChangedInput;
use crate::facts::SkillInvocation;
+use crate::facts::SkillInvocationLocation;
use crate::facts::SkillInvokedInput;
use crate::facts::SubAgentThreadStartedInput;
use crate::facts::TrackEventsContext;
@@ -68,6 +70,7 @@ use codex_protocol::items::TurnItem;
use codex_protocol::protocol::Event;
use codex_protocol::protocol::EventMsg;
use codex_protocol::request_permissions::RequestPermissionsResponse;
+use sha1::Digest;
use std::collections::HashSet;
use std::path::PathBuf;
use std::sync::Arc;
@@ -303,6 +306,51 @@ impl AnalyticsEventsClient {
if invocations.is_empty() {
return;
}
+ for invocation in &invocations {
+ let (location, resource_id, scope) = match &invocation.location {
+ SkillInvocationLocation::Host { path, scope } => (
+ "host",
+ format!(
+ "{:x}",
+ sha1::Sha1::digest(path.to_string_lossy().as_bytes())
+ ),
+ Some(*scope),
+ ),
+ SkillInvocationLocation::Resource { id, scope, .. } => (
+ "resource",
+ format!("{:x}", sha1::Sha1::digest(id.as_bytes())),
+ *scope,
+ ),
+ };
+ let skill_scope = scope
+ .map(|scope| match scope {
+ codex_protocol::protocol::SkillScope::User => "user",
+ codex_protocol::protocol::SkillScope::Repo => "repo",
+ codex_protocol::protocol::SkillScope::System => "system",
+ codex_protocol::protocol::SkillScope::Admin => "admin",
+ })
+ .unwrap_or("unknown");
+ let invoke_type = match invocation.invocation_type {
+ InvocationType::Explicit => "explicit",
+ InvocationType::Implicit => "implicit",
+ };
+ tracing::info!(
+ target: "codex_analytics::skill_invocation",
+ event_name = "codex.skill_invocation",
+ schema_version = 1_u64,
+ thread_id = %tracking.thread_id,
+ turn_id = %tracking.turn_id,
+ skill_name = %invocation.skill_name,
+ skill_resource_id = %resource_id,
+ skill_location = location,
+ skill_scope,
+ plugin_id = invocation.plugin_id.as_deref().unwrap_or("none"),
+ remote_plugin_id = invocation.remote_plugin_id.as_deref().unwrap_or("none"),
+ invocation_type = invoke_type,
+ model = %tracking.model_slug,
+ product_client_id = %tracking.product_client_id,
+ );
+ }
self.record_fact(AnalyticsFact::Custom(CustomAnalyticsFact::SkillInvoked(
SkillInvokedInput {
tracking,
diff --git a/codex-rs/core/tests/suite/skills.rs b/codex-rs/core/tests/suite/skills.rs
index 896ccc73e..20f3e9c64 100644
--- a/codex-rs/core/tests/suite/skills.rs
+++ b/codex-rs/core/tests/suite/skills.rs
@@ -1,6 +1,11 @@
#![allow(clippy::unwrap_used)]
use anyhow::Result;
+use codex_analytics::AnalyticsEventsClient;
+use codex_analytics::InvocationType;
+use codex_analytics::SkillInvocation;
+use codex_analytics::SkillInvocationLocation;
+use codex_analytics::build_track_events_context;
use codex_core::StartIfIdleSubmission;
use codex_core::TurnInput;
use codex_core::TurnInputRequest;
@@ -17,6 +22,7 @@ use codex_protocol::config_types::ModeKind;
use codex_protocol::config_types::Settings;
use codex_protocol::models::PermissionProfile;
use codex_protocol::protocol::AskForApproval;
+use codex_protocol::protocol::SkillScope;
use codex_protocol::protocol::ThreadSettingsOverrides;
use codex_protocol::user_input::UserInput;
use codex_skills_extension::SkillsExtensionConfig;
@@ -39,6 +45,7 @@ use core_test_support::test_codex::turn_permission_fields;
use pretty_assertions::assert_eq;
use std::sync::Arc;
use std::sync::Mutex;
+use tracing_test::traced_test;
#[derive(Default)]
struct SkillInvocationRecorder(Mutex<Vec<(String, SkillInvocationKind)>>);
@@ -191,6 +198,94 @@ async fn user_turn_includes_skill_instructions() -> Result<()> {
Ok(())
}
+#[test]
+#[traced_test]
+fn analytics_client_logs_attributable_explicit_and_implicit_skill_events() {
+ let client = AnalyticsEventsClient::disabled();
+ client.track_skill_invocations(
+ build_track_events_context(
+ "gpt-test".to_string(),
+ "thread-test".to_string(),
+ "turn-test".to_string(),
+ "product-test".to_string(),
+ ),
+ vec![
+ SkillInvocation {
+ skill_name: "host-demo".to_string(),
+ location: SkillInvocationLocation::Host {
+ path: "/tmp/host-demo/SKILL.md".into(),
+ scope: SkillScope::Repo,
+ },
+ plugin_id: None,
+ remote_plugin_id: None,
+ invocation_type: InvocationType::Explicit,
+ },
+ SkillInvocation {
+ skill_name: "resource-demo".to_string(),
+ location: SkillInvocationLocation::Resource {
+ id: "resource://resource-demo/SKILL.md".to_string(),
+ skill_id: Some("resource-secret-id".to_string()),
+ scope: Some(SkillScope::User),
+ },
+ plugin_id: Some("plugin-demo".to_string()),
+ remote_plugin_id: Some("remote-plugin-demo".to_string()),
+ invocation_type: InvocationType::Implicit,
+ },
+ ],
+ );
+
+ logs_assert(|lines: &[&str]| {
+ let field_matches = |line: &str, key: &str, value: &str| {
+ line.contains(&format!("{key}={value}")) || line.contains(&format!("{key}=\"{value}\""))
+ };
+ for (name, location, scope, plugin_id, remote_plugin_id, invoke_type) in [
+ ("host-demo", "host", "repo", "none", "none", "explicit"),
+ (
+ "resource-demo",
+ "resource",
+ "user",
+ "plugin-demo",
+ "remote-plugin-demo",
+ "implicit",
+ ),
+ ] {
+ let line = lines
+ .iter()
+ .find(|line| {
+ line.contains("codex.skill_invocation")
+ && field_matches(line, "skill_name", name)
+ })
+ .ok_or_else(|| format!("missing {invoke_type} {name} skill event"))?;
+ for (key, value) in [
+ ("schema_version", "1"),
+ ("thread_id", "thread-test"),
+ ("turn_id", "turn-test"),
+ ("skill_location", location),
+ ("skill_scope", scope),
+ ("plugin_id", plugin_id),
+ ("remote_plugin_id", remote_plugin_id),
+ ("invocation_type", invoke_type),
+ ("model", "gpt-test"),
+ ("product_client_id", "product-test"),
+ ] {
+ if !field_matches(line, key, value) {
+ return Err(format!("missing {key}={value} in {name} skill event"));
+ }
+ }
+ if !line.contains("skill_resource_id=")
+ || line.contains("/tmp/host-demo")
+ || line.contains("resource://resource-demo")
+ || line.contains("resource-secret-id")
+ {
+ return Err(format!(
+ "resource identity was absent or not opaque for {name}"
+ ));
+ }
+ }
+ Ok(())
+ });
+}
+
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn user_turn_selects_symlinked_skill_by_advertised_discovery_path() -> Result<()> {
skip_if_no_network!(Ok(()));
diff --git a/codex-rs/ext/skills/Cargo.toml b/codex-rs/ext/skills/Cargo.toml
index 59af40c03..c3d9d5114 100644
--- a/codex-rs/ext/skills/Cargo.toml
+++ b/codex-rs/ext/skills/Cargo.toml
@@ -47,3 +47,4 @@ opentelemetry_sdk = { workspace = true }
pretty_assertions = { workspace = true }
tempfile = { workspace = true }
tokio = { workspace = true, features = ["macros", "rt-multi-thread"] }
+tracing-test = { workspace = true, features = ["no-env-filter"] }
diff --git a/codex-rs/ext/skills/src/telemetry.rs b/codex-rs/ext/skills/src/telemetry.rs
index 070cbb976..e87a42e8c 100644
--- a/codex-rs/ext/skills/src/telemetry.rs
+++ b/codex-rs/ext/skills/src/telemetry.rs
@@ -34,10 +34,29 @@ pub(crate) struct SkillTurnMetrics {
#[derive(Default)]
struct TurnUsage {
plugins: HashSet<String>,
+ implicit_resources: HashSet<String>,
failed: bool,
}
impl SkillTurnMetrics {
+ pub(crate) fn new(turn_id: String, model_slug: String, reasoning_effort: String) -> Self {
+ Self {
+ turn_id,
+ model_slug,
+ reasoning_effort,
+ started_at: Instant::now(),
+ usage: Mutex::new(Some(TurnUsage::default())),
+ }
+ }
+
+ pub(crate) fn record_implicit_resource(&self, resource: &str) -> bool {
+ self.usage
+ .lock()
+ .unwrap_or_else(std::sync::PoisonError::into_inner)
+ .as_mut()
+ .is_none_or(|usage| usage.implicit_resources.insert(resource.to_string()))
+ }
+
pub(crate) fn record_plugin(&self, plugin_id: Option<&str>) {
if let Some(usage) = self
.usage
@@ -90,6 +109,24 @@ impl SkillTurnMetrics {
}
}
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn implicit_resource_invocations_are_deduplicated_within_one_turn() {
+ let metrics = SkillTurnMetrics::new(
+ "turn-1".to_string(),
+ "gpt-test".to_string(),
+ "high".to_string(),
+ );
+
+ assert!(metrics.record_implicit_resource("skill://demo/a/SKILL.md"));
+ assert!(!metrics.record_implicit_resource("skill://demo/a/SKILL.md"));
+ assert!(metrics.record_implicit_resource("skill://demo/b/SKILL.md"));
+ }
+}
+
#[derive(Default)]
pub(crate) struct ActiveSkillTurnMetrics(pub(crate) Mutex<Weak<SkillTurnMetrics>>);
@@ -109,13 +146,11 @@ impl TurnLifecycleContributor for SkillTelemetry {
})
.map(|effort| effort.to_string())
.unwrap_or_else(|| "default".to_string());
- input.turn_store.insert(SkillTurnMetrics {
- turn_id: input.turn_id.to_string(),
- model_slug: sanitize_metric_tag_value(input.collaboration_mode.model()),
+ input.turn_store.insert(SkillTurnMetrics::new(
+ input.turn_id.to_string(),
+ sanitize_metric_tag_value(input.collaboration_mode.model()),
reasoning_effort,
- started_at: Instant::now(),
- usage: Mutex::new(Some(TurnUsage::default())),
- });
+ ));
if let Some(turn) = input.turn_store.get::<SkillTurnMetrics>() {
*input
.thread_store
diff --git a/codex-rs/ext/skills/src/tools/mod.rs b/codex-rs/ext/skills/src/tools/mod.rs
index 22bac2bce..2d303416a 100644
--- a/codex-rs/ext/skills/src/tools/mod.rs
+++ b/codex-rs/ext/skills/src/tools/mod.rs
@@ -138,6 +138,15 @@ impl SkillAnalytics {
.unwrap_or_else(std::sync::PoisonError::into_inner)
.upgrade()
.filter(|turn| turn.turn_id == turn_id);
+ if matches!(invocation_type, InvocationType::Implicit)
+ && turn_metrics
+ .as_ref()
+ .is_some_and(|turn| !turn.record_implicit_resource(skill.main_prompt.as_str()))
+ {
+ // The analytics reducer already applies this per-turn resource rule. Suppress here
+ // as well so the always-on local event and extension metric have the same cardinality.
+ return;
+ }
if let Some(turn_metrics) = &turn_metrics {
turn_metrics.record_plugin(skill.plugin_id.as_deref());
}
@@ -386,3 +395,86 @@ fn skill_json_output<T: Serialize>(
SkillToolAuthoritySelector::Executor => Box::new(output),
})
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::catalog::SkillPackageId;
+ use crate::catalog::SkillResourceId;
+ use crate::telemetry::SkillTurnMetrics;
+
+ #[test]
+ #[tracing_test::traced_test]
+ fn repeated_implicit_resource_emits_one_local_event_while_explicit_calls_remain_distinct() {
+ let turn = Arc::new(SkillTurnMetrics::new(
+ "turn-1".to_string(),
+ "gpt-test".to_string(),
+ "high".to_string(),
+ ));
+ let active_turn = Arc::new(ActiveSkillTurnMetrics(std::sync::Mutex::new(
+ Arc::downgrade(&turn),
+ )));
+ let analytics = SkillAnalytics {
+ client: AnalyticsEventsClient::disabled(),
+ metrics: None,
+ active_turn,
+ thread_id: "thread-1".to_string(),
+ product_client_id: "product-test".to_string(),
+ };
+ let main_resource = "skill://demo/dedupe/SKILL.md";
+ let skill = SkillCatalogEntry::new(
+ SkillPackageId("skill://demo/dedupe".to_string()),
+ SkillAuthority::new(SkillSourceKind::Orchestrator, CODEX_APPS_MCP_SERVER_NAME),
+ "demo:dedupe",
+ "Dedupe telemetry test.",
+ SkillResourceId::new(main_resource),
+ );
+
+ for invocation_type in [
+ InvocationType::Implicit,
+ InvocationType::Implicit,
+ InvocationType::Explicit,
+ InvocationType::Explicit,
+ ] {
+ analytics.track_skill_invocation(
+ &skill,
+ "gpt-test".to_string(),
+ "turn-1".to_string(),
+ invocation_type,
+ );
+ }
+
+ logs_assert(|lines: &[&str]| {
+ let field_matches = |line: &str, key: &str, value: &str| {
+ line.contains(&format!("{key}={value}"))
+ || line.contains(&format!("{key}=\"{value}\""))
+ };
+ let matching = |invocation_type: &str| {
+ lines
+ .iter()
+ .filter(|line| {
+ line.contains("codex.skill_invocation")
+ && field_matches(line, "skill_name", "demo:dedupe")
+ && field_matches(line, "invocation_type", invocation_type)
+ })
+ .copied()
+ .collect::<Vec<_>>()
+ };
+ let implicit = matching("implicit");
+ let explicit = matching("explicit");
+ if implicit.len() != 1 || explicit.len() != 2 {
+ return Err(format!(
+ "expected one implicit and two explicit local events, got implicit={implicit:?} explicit={explicit:?}"
+ ));
+ }
+ if implicit
+ .iter()
+ .chain(explicit.iter())
+ .any(|line| line.contains(main_resource))
+ {
+ return Err("local skill event exposed the raw main resource".to_string());
+ }
+ Ok(())
+ });
+ }
+}
Current-main patch (SHA-256 957762a6…f72f)
diff --git a/codex-rs/analytics/src/client.rs b/codex-rs/analytics/src/client.rs
index 7b5056acc..2963a8692 100644
--- a/codex-rs/analytics/src/client.rs
+++ b/codex-rs/analytics/src/client.rs
@@ -18,6 +18,7 @@ use crate::facts::ExternalAgentConfigImportFailureInput;
use crate::facts::HookRunFact;
use crate::facts::HookRunInput;
use crate::facts::ImagePreparationFact;
+use crate::facts::InvocationType;
use crate::facts::PluginInstallFailedInput;
use crate::facts::PluginInstallRequested;
use crate::facts::PluginInstallRequestedInput;
@@ -26,6 +27,7 @@ use crate::facts::PluginMeasurementsInput;
use crate::facts::PluginState;
use crate::facts::PluginStateChangedInput;
use crate::facts::SkillInvocation;
+use crate::facts::SkillInvocationLocation;
use crate::facts::SkillInvokedInput;
use crate::facts::SubAgentThreadStartedInput;
use crate::facts::TrackEventsContext;
@@ -68,6 +70,7 @@ use codex_protocol::items::TurnItem;
use codex_protocol::protocol::Event;
use codex_protocol::protocol::EventMsg;
use codex_protocol::request_permissions::RequestPermissionsResponse;
+use sha1::Digest;
use std::collections::HashSet;
use std::path::PathBuf;
use std::sync::Arc;
@@ -303,6 +306,51 @@ impl AnalyticsEventsClient {
if invocations.is_empty() {
return;
}
+ for invocation in &invocations {
+ let (location, resource_id, scope) = match &invocation.location {
+ SkillInvocationLocation::Host { path, scope } => (
+ "host",
+ format!(
+ "{:x}",
+ sha1::Sha1::digest(path.to_string_lossy().as_bytes())
+ ),
+ Some(*scope),
+ ),
+ SkillInvocationLocation::Resource { id, scope, .. } => (
+ "resource",
+ format!("{:x}", sha1::Sha1::digest(id.as_bytes())),
+ *scope,
+ ),
+ };
+ let skill_scope = scope
+ .map(|scope| match scope {
+ codex_protocol::protocol::SkillScope::User => "user",
+ codex_protocol::protocol::SkillScope::Repo => "repo",
+ codex_protocol::protocol::SkillScope::System => "system",
+ codex_protocol::protocol::SkillScope::Admin => "admin",
+ })
+ .unwrap_or("unknown");
+ let invoke_type = match invocation.invocation_type {
+ InvocationType::Explicit => "explicit",
+ InvocationType::Implicit => "implicit",
+ };
+ tracing::info!(
+ target: "codex_analytics::skill_invocation",
+ event_name = "codex.skill_invocation",
+ schema_version = 1_u64,
+ thread_id = %tracking.thread_id,
+ turn_id = %tracking.turn_id,
+ skill_name = %invocation.skill_name,
+ skill_resource_id = %resource_id,
+ skill_location = location,
+ skill_scope,
+ plugin_id = invocation.plugin_id.as_deref().unwrap_or("none"),
+ remote_plugin_id = invocation.remote_plugin_id.as_deref().unwrap_or("none"),
+ invocation_type = invoke_type,
+ model = %tracking.model_slug,
+ product_client_id = %tracking.product_client_id,
+ );
+ }
self.record_fact(AnalyticsFact::Custom(CustomAnalyticsFact::SkillInvoked(
SkillInvokedInput {
tracking,
diff --git a/codex-rs/core/tests/suite/skills.rs b/codex-rs/core/tests/suite/skills.rs
index 896ccc73e..20f3e9c64 100644
--- a/codex-rs/core/tests/suite/skills.rs
+++ b/codex-rs/core/tests/suite/skills.rs
@@ -1,6 +1,11 @@
#![allow(clippy::unwrap_used)]
use anyhow::Result;
+use codex_analytics::AnalyticsEventsClient;
+use codex_analytics::InvocationType;
+use codex_analytics::SkillInvocation;
+use codex_analytics::SkillInvocationLocation;
+use codex_analytics::build_track_events_context;
use codex_core::StartIfIdleSubmission;
use codex_core::TurnInput;
use codex_core::TurnInputRequest;
@@ -17,6 +22,7 @@ use codex_protocol::config_types::ModeKind;
use codex_protocol::config_types::Settings;
use codex_protocol::models::PermissionProfile;
use codex_protocol::protocol::AskForApproval;
+use codex_protocol::protocol::SkillScope;
use codex_protocol::protocol::ThreadSettingsOverrides;
use codex_protocol::user_input::UserInput;
use codex_skills_extension::SkillsExtensionConfig;
@@ -39,6 +45,7 @@ use core_test_support::test_codex::turn_permission_fields;
use pretty_assertions::assert_eq;
use std::sync::Arc;
use std::sync::Mutex;
+use tracing_test::traced_test;
#[derive(Default)]
struct SkillInvocationRecorder(Mutex<Vec<(String, SkillInvocationKind)>>);
@@ -191,6 +198,94 @@ async fn user_turn_includes_skill_instructions() -> Result<()> {
Ok(())
}
+#[test]
+#[traced_test]
+fn analytics_client_logs_attributable_explicit_and_implicit_skill_events() {
+ let client = AnalyticsEventsClient::disabled();
+ client.track_skill_invocations(
+ build_track_events_context(
+ "gpt-test".to_string(),
+ "thread-test".to_string(),
+ "turn-test".to_string(),
+ "product-test".to_string(),
+ ),
+ vec![
+ SkillInvocation {
+ skill_name: "host-demo".to_string(),
+ location: SkillInvocationLocation::Host {
+ path: "/tmp/host-demo/SKILL.md".into(),
+ scope: SkillScope::Repo,
+ },
+ plugin_id: None,
+ remote_plugin_id: None,
+ invocation_type: InvocationType::Explicit,
+ },
+ SkillInvocation {
+ skill_name: "resource-demo".to_string(),
+ location: SkillInvocationLocation::Resource {
+ id: "resource://resource-demo/SKILL.md".to_string(),
+ skill_id: Some("resource-secret-id".to_string()),
+ scope: Some(SkillScope::User),
+ },
+ plugin_id: Some("plugin-demo".to_string()),
+ remote_plugin_id: Some("remote-plugin-demo".to_string()),
+ invocation_type: InvocationType::Implicit,
+ },
+ ],
+ );
+
+ logs_assert(|lines: &[&str]| {
+ let field_matches = |line: &str, key: &str, value: &str| {
+ line.contains(&format!("{key}={value}")) || line.contains(&format!("{key}=\"{value}\""))
+ };
+ for (name, location, scope, plugin_id, remote_plugin_id, invoke_type) in [
+ ("host-demo", "host", "repo", "none", "none", "explicit"),
+ (
+ "resource-demo",
+ "resource",
+ "user",
+ "plugin-demo",
+ "remote-plugin-demo",
+ "implicit",
+ ),
+ ] {
+ let line = lines
+ .iter()
+ .find(|line| {
+ line.contains("codex.skill_invocation")
+ && field_matches(line, "skill_name", name)
+ })
+ .ok_or_else(|| format!("missing {invoke_type} {name} skill event"))?;
+ for (key, value) in [
+ ("schema_version", "1"),
+ ("thread_id", "thread-test"),
+ ("turn_id", "turn-test"),
+ ("skill_location", location),
+ ("skill_scope", scope),
+ ("plugin_id", plugin_id),
+ ("remote_plugin_id", remote_plugin_id),
+ ("invocation_type", invoke_type),
+ ("model", "gpt-test"),
+ ("product_client_id", "product-test"),
+ ] {
+ if !field_matches(line, key, value) {
+ return Err(format!("missing {key}={value} in {name} skill event"));
+ }
+ }
+ if !line.contains("skill_resource_id=")
+ || line.contains("/tmp/host-demo")
+ || line.contains("resource://resource-demo")
+ || line.contains("resource-secret-id")
+ {
+ return Err(format!(
+ "resource identity was absent or not opaque for {name}"
+ ));
+ }
+ }
+ Ok(())
+ });
+}
+
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn user_turn_selects_symlinked_skill_by_advertised_discovery_path() -> Result<()> {
skip_if_no_network!(Ok(()));
diff --git a/codex-rs/ext/skills/Cargo.toml b/codex-rs/ext/skills/Cargo.toml
index 59af40c03..c3d9d5114 100644
--- a/codex-rs/ext/skills/Cargo.toml
+++ b/codex-rs/ext/skills/Cargo.toml
@@ -47,3 +47,4 @@ opentelemetry_sdk = { workspace = true }
pretty_assertions = { workspace = true }
tempfile = { workspace = true }
tokio = { workspace = true, features = ["macros", "rt-multi-thread"] }
+tracing-test = { workspace = true, features = ["no-env-filter"] }
diff --git a/codex-rs/ext/skills/src/telemetry.rs b/codex-rs/ext/skills/src/telemetry.rs
index 070cbb976..e87a42e8c 100644
--- a/codex-rs/ext/skills/src/telemetry.rs
+++ b/codex-rs/ext/skills/src/telemetry.rs
@@ -34,10 +34,29 @@ pub(crate) struct SkillTurnMetrics {
#[derive(Default)]
struct TurnUsage {
plugins: HashSet<String>,
+ implicit_resources: HashSet<String>,
failed: bool,
}
impl SkillTurnMetrics {
+ pub(crate) fn new(turn_id: String, model_slug: String, reasoning_effort: String) -> Self {
+ Self {
+ turn_id,
+ model_slug,
+ reasoning_effort,
+ started_at: Instant::now(),
+ usage: Mutex::new(Some(TurnUsage::default())),
+ }
+ }
+
+ pub(crate) fn record_implicit_resource(&self, resource: &str) -> bool {
+ self.usage
+ .lock()
+ .unwrap_or_else(std::sync::PoisonError::into_inner)
+ .as_mut()
+ .is_none_or(|usage| usage.implicit_resources.insert(resource.to_string()))
+ }
+
pub(crate) fn record_plugin(&self, plugin_id: Option<&str>) {
if let Some(usage) = self
.usage
@@ -90,6 +109,24 @@ impl SkillTurnMetrics {
}
}
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn implicit_resource_invocations_are_deduplicated_within_one_turn() {
+ let metrics = SkillTurnMetrics::new(
+ "turn-1".to_string(),
+ "gpt-test".to_string(),
+ "high".to_string(),
+ );
+
+ assert!(metrics.record_implicit_resource("skill://demo/a/SKILL.md"));
+ assert!(!metrics.record_implicit_resource("skill://demo/a/SKILL.md"));
+ assert!(metrics.record_implicit_resource("skill://demo/b/SKILL.md"));
+ }
+}
+
#[derive(Default)]
pub(crate) struct ActiveSkillTurnMetrics(pub(crate) Mutex<Weak<SkillTurnMetrics>>);
@@ -109,13 +146,11 @@ impl TurnLifecycleContributor for SkillTelemetry {
})
.map(|effort| effort.to_string())
.unwrap_or_else(|| "default".to_string());
- input.turn_store.insert(SkillTurnMetrics {
- turn_id: input.turn_id.to_string(),
- model_slug: sanitize_metric_tag_value(input.collaboration_mode.model()),
+ input.turn_store.insert(SkillTurnMetrics::new(
+ input.turn_id.to_string(),
+ sanitize_metric_tag_value(input.collaboration_mode.model()),
reasoning_effort,
- started_at: Instant::now(),
- usage: Mutex::new(Some(TurnUsage::default())),
- });
+ ));
if let Some(turn) = input.turn_store.get::<SkillTurnMetrics>() {
*input
.thread_store
diff --git a/codex-rs/ext/skills/src/tools/mod.rs b/codex-rs/ext/skills/src/tools/mod.rs
index 22bac2bce..2d303416a 100644
--- a/codex-rs/ext/skills/src/tools/mod.rs
+++ b/codex-rs/ext/skills/src/tools/mod.rs
@@ -138,6 +138,15 @@ impl SkillAnalytics {
.unwrap_or_else(std::sync::PoisonError::into_inner)
.upgrade()
.filter(|turn| turn.turn_id == turn_id);
+ if matches!(invocation_type, InvocationType::Implicit)
+ && turn_metrics
+ .as_ref()
+ .is_some_and(|turn| !turn.record_implicit_resource(skill.main_prompt.as_str()))
+ {
+ // The analytics reducer already applies this per-turn resource rule. Suppress here
+ // as well so the always-on local event and extension metric have the same cardinality.
+ return;
+ }
if let Some(turn_metrics) = &turn_metrics {
turn_metrics.record_plugin(skill.plugin_id.as_deref());
}
@@ -386,3 +395,86 @@ fn skill_json_output<T: Serialize>(
SkillToolAuthoritySelector::Executor => Box::new(output),
})
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::catalog::SkillPackageId;
+ use crate::catalog::SkillResourceId;
+ use crate::telemetry::SkillTurnMetrics;
+
+ #[test]
+ #[tracing_test::traced_test]
+ fn repeated_implicit_resource_emits_one_local_event_while_explicit_calls_remain_distinct() {
+ let turn = Arc::new(SkillTurnMetrics::new(
+ "turn-1".to_string(),
+ "gpt-test".to_string(),
+ "high".to_string(),
+ ));
+ let active_turn = Arc::new(ActiveSkillTurnMetrics(std::sync::Mutex::new(
+ Arc::downgrade(&turn),
+ )));
+ let analytics = SkillAnalytics {
+ client: AnalyticsEventsClient::disabled(),
+ metrics: None,
+ active_turn,
+ thread_id: "thread-1".to_string(),
+ product_client_id: "product-test".to_string(),
+ };
+ let main_resource = "skill://demo/dedupe/SKILL.md";
+ let skill = SkillCatalogEntry::new(
+ SkillPackageId("skill://demo/dedupe".to_string()),
+ SkillAuthority::new(SkillSourceKind::Orchestrator, CODEX_APPS_MCP_SERVER_NAME),
+ "demo:dedupe",
+ "Dedupe telemetry test.",
+ SkillResourceId::new(main_resource),
+ );
+
+ for invocation_type in [
+ InvocationType::Implicit,
+ InvocationType::Implicit,
+ InvocationType::Explicit,
+ InvocationType::Explicit,
+ ] {
+ analytics.track_skill_invocation(
+ &skill,
+ "gpt-test".to_string(),
+ "turn-1".to_string(),
+ invocation_type,
+ );
+ }
+
+ logs_assert(|lines: &[&str]| {
+ let field_matches = |line: &str, key: &str, value: &str| {
+ line.contains(&format!("{key}={value}"))
+ || line.contains(&format!("{key}=\"{value}\""))
+ };
+ let matching = |invocation_type: &str| {
+ lines
+ .iter()
+ .filter(|line| {
+ line.contains("codex.skill_invocation")
+ && field_matches(line, "skill_name", "demo:dedupe")
+ && field_matches(line, "invocation_type", invocation_type)
+ })
+ .copied()
+ .collect::<Vec<_>>()
+ };
+ let implicit = matching("implicit");
+ let explicit = matching("explicit");
+ if implicit.len() != 1 || explicit.len() != 2 {
+ return Err(format!(
+ "expected one implicit and two explicit local events, got implicit={implicit:?} explicit={explicit:?}"
+ ));
+ }
+ if implicit
+ .iter()
+ .chain(explicit.iter())
+ .any(|line| line.contains(main_resource))
+ {
+ return Err("local skill event exposed the raw main resource".to_string());
+ }
+ Ok(())
+ });
+ }
+}
Verification
Exact release:
- Direct disabled-client explicit+implicit event/privacy test: 1/1 passed.
- Skill extension unit/integration suites: 175/175 passed.
codex-analytics: 103/103 passed.- Repeated same-turn resource test observed exactly one implicit local event,
two distinct explicit events, and no raw resource URI. - Analytics/extension clippy, formatter, and diff-check passed.
codex-clirelease build passed in 17m36s; binary reports 0.151.0 and contains the event token.- One workspace-wide attempt was blocked before test execution by the missing
rusty_v8 v150.4.0macOS arm64 archive (HTTP 404); it was not reported as a passing suite.
Current main:
- Direct event/privacy test: 1/1 passed.
- Skill extension unit/integration suites: 175/175 passed.
codex-analytics: 103/103 passed.- Analytics/extension clippy, formatter, and clean worktree passed.
Local installation proof
Local prototype evidence after the independent source review passed:
- Installed only the standalone 0.151.0 primary; candidate SHA-256
4ad4cf5087725b6e3d292b370fa1123e5a8e074212ef86df59a5f32b1d2d9212, arm64, strict-valid ad-hoc runtime signature, identifiercodex, no TeamIdentifier, with the two original entitlements. - Retained the exact official preimage at mode 0600: SHA-256
98491713ffb196061003ee148636e743997cc31d76144ba7c53462269896891d. A disposable recovery copy read back 0.151.0, saved ChatGPT login, TeamIdentifier2DC432GLL2, and both entitlements. - Started the installed binary's app-server, created one ephemeral
gpt-5.6-luna/lowturn, and explicitly invokedopenai-docs. logs_2.sqliterow59056923persistedtarget=codex_analytics::skill_invocation,event_name=codex.skill_invocation,schema_version=1, exact thread/turn,skill_name=openai-docs,invocation_type=explicit, model, and product client. Querying by target/thread/turn/skill returned exactly one row; raw Skill-path hits were zero.- Code-mode host, package manifest, current symlink, ChatGPT.app fallback, config, and auth hashes were unchanged.
- The first smoke request was rejected before a turn because the harness used obsolete sandbox enum
readOnly; correcting only that request toread-onlyproduced the passing readback above. This is pass-on-retry, not first-pass success.
This local installation is evidence for the proposed approach, not a claim that official Codex currently ships the event.
Notes on duplicates and scope
- #39906 would let external hook handlers observe skill invocation. That is useful but does not provide a default local audit row when hooks are absent/disabled.
- #35650 enriches a remote aggregate metric and cannot bind a local row to a specific thread/turn.
- #35869 addresses an extension metric emission gap, not the local tracing surface.
No external PR is proposed; the repository's current contribution policy asks external contributors to share reproduction, root-cause analysis, logs, and potential approaches through issues.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with codex-rs/analytics/src/client.rs and the existing track_skill_invocations boundary, then read codex-rs/ext/skills/src/telemetry.rs and src/tools/mod.rs for the two caller paths. Run the direct test in codex-rs/core/tests/suite/skills.rs first. Done means attributable, privacy-bounded events are emitted even with remote analytics disabled, with the documented implicit-invocation cardinality and test coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100