getsentry / getsentry/sentry

gen_ai cost metrics double-count hierarchical spans

Open
#114,914 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
44.8k
Forks
4.9k
Avg merge
21h 10m
Merged PRs (30d)
635

Description

`sum(gen_ai.cost.total_tokens)` over-reports actual LLM spend by 2× (or more) because Relay computes cost attributes independently on every AI span, including hierarchical parent-child pairs that carry identical token counts.

## Root cause

The OpenTelemetry GenAI semantic conventions (`Development` status) recommend `gen_ai.usage.{input,output}_tokens` on **both** agent-level spans (`gen_ai.invoke_agent`) and individual inference spans (`gen_ai.generate_text` / `gen_ai.chat`). The intended semantics are:

- **`invoke_agent`** (parent): aggregate token usage across all LLM calls in the agent invocation
- **`generate_text` / `chat`** (child): per-call token usage

When an agent makes a single LLM call, both spans carry **identical** token counts — the aggregate equals the individual. This is spec-compliant behavior; the OTel GenAI semconv has no guidance on deduplication across hierarchical spans.

Relay's cost normalization (`relay-event-normalization` `eap/ai.rs` / `normalize/span/ai.rs`) computes `gen_ai.cost.*` on every span that has token usage attributes, without awareness of parent-child relationships. Any `sum()` query across all AI spans then double-counts.

**Verified against live data** (mcp-server project, last 5 days):

- `sum(gen_ai.cost.total_tokens)` across all spans: **~$9,900**
- `sum(gen_ai.cost.total_tokens)` filtered to `gen_ai.invoke_agent` only: **~$4,949**
- `sum(gen_ai.cost.total_tokens)` filtered to `gen_ai.generate_text` only: **~$4,948**
- Actual OpenAI billing for the same period: **~$4,500**

The unfiltered sum is ~2× the real cost. For multi-step agents (multiple child LLM calls per parent), the over-count would be `n+1` for `n` child spans if the parent aggregates correctly.

## Where it matters in the product

- **AI Conversations** view — any cost aggregation across conversations
- **Explore / Spans** — `sum(gen_ai.cost.total_tokens)` queries
- Any dashboard or alert built on `gen_ai.cost.*` span attributes

## SDK context

The `sentry-javascript` Vercel AI integration (`vercelAIIntegration`) maps AI SDK telemetry to Sentry spans:

- `ai.generateText` / `ai.streamText` → `gen_ai.invoke_agent` (parent)
- `ai.generateText.doGenerate` / `ai.streamText.doStream` → `gen_ai.generate_text` (child)

Both carry `gen_ai.usage.*` per the OTel spec. This will affect every customer using the Vercel AI SDK integration (and likely any OTel-compliant AI instrumentation that emits hierarchical spans).

## Options

- **Query-level fix**: default AI cost queries to leaf spans only (`gen_ai.generate_text`, `gen_ai.chat`), excluding parent agent spans from aggregation. Simplest, but fragile if span hierarchies vary.
- **Relay-level dedup**: skip computing `gen_ai.cost.*` on parent spans when child spans already carry token usage. Requires hierarchy awareness in normalization.
- **Product-level dedup**: the AI module's aggregation queries exclude known parent ops, or the product uses a dedicated "deduplicated cost" metric derived from leaf spans only.

The OTel GenAI semconv is still in `Development` status and the agent spans spec is new. This is a known gap in the upstream spec — there's no guidance on how platforms should aggregate costs across hierarchical spans. Worth raising in `open-telemetry/semantic-conventions` as well.

Action taken on behalf of David Cramer.

Contributor guide

Open the contributing guide

Research direction

Start with relay-event-normalization/eap/ai.rs and normalize/span/ai.rs, then trace how the sentry-javascript vercelAIIntegration creates invoke_agent and generate_text spans. Compare the query-level, Relay-level, and product-level options before choosing an approach. Done means hierarchical spans no longer cause gen_ai.cost.* aggregations to double-count token usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, rust
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.