gen_ai cost metrics double-count hierarchical spans
- Dominant language
- Python
- Stars
- 44.8k
- Forks
- 4.9k
- Avg merge
- 21h 10m
- Merged PRs (30d)
- 635
Description
`sum(gen_ai.cost.total_tokens)` over-reports actual LLM spend by 2× (or more) because Relay computes cost attributes independently on every AI span, including hierarchical parent-child pairs that carry identical token counts.
## Root cause
The OpenTelemetry GenAI semantic conventions (`Development` status) recommend `gen_ai.usage.{input,output}_tokens` on **both** agent-level spans (`gen_ai.invoke_agent`) and individual inference spans (`gen_ai.generate_text` / `gen_ai.chat`). The intended semantics are:
- **`invoke_agent`** (parent): aggregate token usage across all LLM calls in the agent invocation
- **`generate_text` / `chat`** (child): per-call token usage
When an agent makes a single LLM call, both spans carry **identical** token counts — the aggregate equals the individual. This is spec-compliant behavior; the OTel GenAI semconv has no guidance on deduplication across hierarchical spans.
Relay's cost normalization (`relay-event-normalization` `eap/ai.rs` / `normalize/span/ai.rs`) computes `gen_ai.cost.*` on every span that has token usage attributes, without awareness of parent-child relationships. Any `sum()` query across all AI spans then double-counts.
**Verified against live data** (mcp-server project, last 5 days):
- `sum(gen_ai.cost.total_tokens)` across all spans: **~$9,900**
- `sum(gen_ai.cost.total_tokens)` filtered to `gen_ai.invoke_agent` only: **~$4,949**
- `sum(gen_ai.cost.total_tokens)` filtered to `gen_ai.generate_text` only: **~$4,948**
- Actual OpenAI billing for the same period: **~$4,500**
The unfiltered sum is ~2× the real cost. For multi-step agents (multiple child LLM calls per parent), the over-count would be `n+1` for `n` child spans if the parent aggregates correctly.
## Where it matters in the product
- **AI Conversations** view — any cost aggregation across conversations
- **Explore / Spans** — `sum(gen_ai.cost.total_tokens)` queries
- Any dashboard or alert built on `gen_ai.cost.*` span attributes
## SDK context
The `sentry-javascript` Vercel AI integration (`vercelAIIntegration`) maps AI SDK telemetry to Sentry spans:
- `ai.generateText` / `ai.streamText` → `gen_ai.invoke_agent` (parent)
- `ai.generateText.doGenerate` / `ai.streamText.doStream` → `gen_ai.generate_text` (child)
Both carry `gen_ai.usage.*` per the OTel spec. This will affect every customer using the Vercel AI SDK integration (and likely any OTel-compliant AI instrumentation that emits hierarchical spans).
## Options
- **Query-level fix**: default AI cost queries to leaf spans only (`gen_ai.generate_text`, `gen_ai.chat`), excluding parent agent spans from aggregation. Simplest, but fragile if span hierarchies vary.
- **Relay-level dedup**: skip computing `gen_ai.cost.*` on parent spans when child spans already carry token usage. Requires hierarchy awareness in normalization.
- **Product-level dedup**: the AI module's aggregation queries exclude known parent ops, or the product uses a dedicated "deduplicated cost" metric derived from leaf spans only.
The OTel GenAI semconv is still in `Development` status and the agent spans spec is new. This is a known gap in the upstream spec — there's no guidance on how platforms should aggregate costs across hierarchical spans. Worth raising in `open-telemetry/semantic-conventions` as well.
Action taken on behalf of David Cramer.
Contributor guide
Research direction
Start with relay-event-normalization/eap/ai.rs and normalize/span/ai.rs, then trace how the sentry-javascript vercelAIIntegration creates invoke_agent and generate_text spans. Compare the query-level, Relay-level, and product-level options before choosing an approach. Done means hierarchical spans no longer cause gen_ai.cost.* aggregations to double-count token usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, rust
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100