ADORSYS-GIS / ADORSYS-GIS/lightbridge-governance

[Epic]: Per-user usage for Claude Code and Codex via native OTLP push

Open
#30 1 comment 0 reactions 1 assignee Claimed by @Motouom View on GitHub
epic governance
Dominant language
Rust
Stars
1
Forks
2
Avg merge
13h 13m
Merged PRs (30d)
110

Description

### Executive Summary

We want **per-user usage and cost visibility for Claude Code and OpenAI Codex**, because both are real, paid AI usage by our engineers that is currently invisible to every governance surface we have. Gateway spend is measured to the micro-dollar; LibreChat and opencode are attributed per user; Copilot has a connector being built. Claude Code and Codex have nothing.

This epic exists to close that gap — and the research says it is far cheaper than it looks, because **both products ship native OpenTelemetry exporters that stamp `user.email`**, and the ingest endpoint they would push to is *already designed* for the Foundry connector.

### Strategic Intent

Make Claude Code and Codex usage as visible as gateway usage, by generalising one OTLP ingest endpoint we already decided to build rather than writing two more vendor-specific connectors.

### Problem Statement

- **Users / governance team:** cannot answer "what are we spending on Claude Code and Codex, and who is spending it?" There is no data anywhere in our systems.
- **Developers / operations:** the usage exists in each vendor's console, per-vendor, un-joinable to our identities, and un-joinable to the gateway spend we already measure.
- **Business:** total AI cost per engineer cannot be stated while two of the most-used tools are missing from it.
- ⚠️ **Security / privacy, today, independent of this epic:** Codex's `metrics_exporter` **defaults to `statsig`** — metrics ship to OpenAI unless explicitly disabled. Nobody chose that.

### Desired Outcome

- Claude Code and Codex sessions land as per-user, per-model, per-tool records in the governance store, in the same integer micro-USD unit as gateway spend.
- The config that makes that happen is **distributed centrally**, not left to each developer — Claude Code's is admin-enforceable.
- Total AI spend per engineer is answerable across gateway + Copilot + Claude Code + Codex in one Grafana view.
- Codex stops sending metrics to OpenAI by default.

### Scope (In / Out)

#### In Scope
- Generalising the RFC-0002 OTLP ingest from "the Foundry connector" to **"the push connector"**, with per-provider normalizers.
- `claude_code.*` and Codex session events → the shared execution / model-call / tool-call records.
- Per-developer integration tokens issued by the existing registry; `identity_maps` rows binding them to the Keycloak `sub`.
- Central config distribution: Claude Code **managed settings** (developers cannot override), Codex `[otel]` in `~/.codex/config.toml`. Coder workspace template first, dotfiles second.
- Turning Codex's `statsig` metrics exporter **off**.

#### Out of Scope
- **Routing either tool through our gateway.** That is authentication work and already has tickets — ADORSYS-GIS/ai-helm#680 (Claude Code) and #679 (Codex). This epic deliberately does **not** widen them; see below.
- **Content capture.** `OTEL_LOG_USER_PROMPTS` / `log_user_prompt` stay off. RFC-0002's `metadata_only` default applies verbatim — token counts need none of it.
- **Vendor usage-API pull connectors.** Optional later reconciliation, and only if we buy Console/Enterprise seats. Never the system of record.
- Anthropic's own self-hosted `claude gateway`. Interesting (Keycloak OIDC, free, best per-user attribution in the report) but its upstream enum is closed to five named providers and cannot reach our fleet. Separate ADR if ever wanted.

### Why OTLP push and not the obvious routes

The brief posed two routes; the research surfaced four and ranked them:

| Route | Claude Code | Codex |
|---|---|---|
| Gateway client | Viable, blocked on a backend-schema question | Viable, blocked on a `/v1/responses` question |
| Vendor usage-API pull | Per-user, daily, free — but blind to subscription users | Enterprise/Edu only for ChatGPT seats |
| **Native client OTLP push** ⭐ | First-class, **admin-enforceable** | First-class |
| Anthropic `claude gateway` | Free, Keycloak OIDC — but cannot reach our fleet | n/a |

OTLP push wins because it is provider-agnostic, **does not touch the request path**, works whether or not the tools ever point at our gateway, needs no vendor commercial relationship, and the hard part — public TLS host, AuthConfig #3, collector, quota — is **already designed and decided** in ADR-0006 / RFC-0002.

### Source of truth (links)

- **The investigation:** `docs/rfc/sources/claude-codex-usage-investigation.md` (#29). 61 claims marked verified with vendor URLs or `file:line`; 22 marked explicitly unconfirmed.
- Builds on: this repo's ADR-0006 and RFC-0002 (the OTLP ingest), ADR-0008 (integer micro-USD), the registry epic #4
- Related but distinct: ADORSYS-GIS/ai-helm#680, ADORSYS-GIS/ai-helm#679 (the *auth* tickets)
- Precedent for the pull-connector shape, if Phase C ever happens: #5 (Copilot)

### Key Assumptions

1. ⚠️ **Both exporters emit enough to compute cost.** Verified they emit token counts; that pricing can be applied per model on our side is assumed. **[?] Codex `exec` still does not export `codex.turn.token_usage`** ([openai/codex#33668](https://github.com/openai/codex/issues/33668), open) — token counts appear only as span/log attributes there.
2. **Managed settings can pin Claude Code's telemetry.** Verified for Claude Code. ⚠️ **[?] whether Codex's admin `requirements.toml` can pin `[otel]` is unconfirmed** — if it cannot, Codex config is advisory and a developer can turn it off.
3. **Per-developer ingest tokens are the identity anchor, not payload identity.** ⚠️ Both products' `user.email` is populated only under *vendor* sign-in and is `None` under API-key/custom-provider auth. Cross-check the emitted email against the token's subject and **alert on mismatch rather than trusting it**.
4. LGTM stays single-tenant (decision 3 in the ai-helm roadmap); `X-Scope-OrgID` remains inert.

### Constraints

- ⚠️ **Anthropic prohibits routing Free/Pro/Max subscription credentials through a gateway**, and reserves enforcement "without prior notice". Constrains #680, not this epic — but it is the finding most likely to sink the gateway route, so it is recorded here.
- Content capture off; `metadata_only` default.
- Integer micro-USD everywhere.
- Values-repo-first for anything reaching `ai-helm-values`.

### Risks

| Risk | Probability | Impact | Mitigation |
|---|---|---|---|
| ⚠️ **Codex ships metrics to OpenAI by default (`statsig`)** | **Certain today** | Medium | Fix it in the config rollout; it is a finding in its own right |
| Codex telemetry is advisory (no admin pin) and developers disable it | Medium | High | Confirm the `requirements.toml` question early; if it cannot be pinned, coverage is best-effort and must be *reported* as such, not assumed |
| `codex exec` token counts missing (#33668) | **Confirmed open** | Medium | Derive from span attributes; do not assume the metric exists |
| Identity mismatch between token subject and emitted email | Medium | High | Alert on mismatch; never overwrite the token-derived identity with payload identity |
| A public OTLP endpoint is abused | Medium | Medium | Per-integration revocable tokens + quota, exactly as RFC-0002 specifies |
| Cost figures drift from the vendor's own books | Medium | Low | We price tokens ourselves; treat vendor analytics as cross-check if ever added |

### Success metrics

| Metric | Current | Target | Source |
|---|---|---|---|
| Claude Code sessions attributed to a user | 0 | ~all, from enrolled developers | governance store |
| Codex sessions attributed to a user | 0 | ~all, from enrolled developers | governance store |
| Codex installs still exporting metrics to OpenAI | **all** | **0** | config rollout |
| Engineers whose total AI spend is answerable in one view | partial | all four sources joined | Grafana |
| New vendor-specific connectors written | — | **0** (one generalised push connector) | repo |

### Child User Stories

- Generalise the OTLP ingest to a provider-agnostic push connector *(the 80/20)*
- Claude Code normalizer + managed-settings rollout
- Codex normalizer + `[otel]` rollout, **including disabling the `statsig` default**
- Per-developer ingest tokens + `identity_maps` binding, with mismatch alerting
- Grafana: total AI spend per engineer across all four sources
- Spike: can Codex's admin `requirements.toml` pin `[otel]`? *(gates how much coverage we can promise)*

### Human accountable owner

@stephane-segning

### AI Usage Declaration

Drafting, Research, Ticket decomposition, Technical proposal

### Human verification completed

- [x] Intent checked against source of truth
- [x] Risks reviewed
- [x] No unverified AI claim remains — every uncertain item above is marked ⚠️ **[?]** and carries its consequence
- [x] I am the accountable owner and accept responsibility for this epic.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.