traceloop / traceloop/openllmetry

Same cached Claude call via anthropic vs bedrock packages emits different gen_ai.usage.input_tokens — which is intended?

Open
#4,449 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.4k
Forks
1.1k
Avg merge
8d 14h
Merged PRs (30d)
2

Description

A cross-package semantics question found while auditing cache-token accounting against the OTel GenAI conventions.

The observation

For the same cached Claude call, two instrumentation packages in this repo emit different gen_ai.usage.input_tokens:

  • anthropic package folds cache in: packages/opentelemetry-instrumentation-anthropic/.../anthropic/__init__.py (~L226-234): input_tokens = prompt_tokens + cache_read_tokens + cache_creation_tokens, with cache subset attributes emitted alongside (streaming path matches, streaming.py ~L85-105).
  • bedrock package does not: packages/opentelemetry-instrumentation-bedrock/.../bedrock/span_utils.py (converse_usage_record, ~L1071-1099): input_tokens = usage.inputTokens as-is, with cacheReadInputTokens / cacheWriteInputTokens emitted only as separate attributes.

For Claude-on-Bedrock via Converse, the provider's inputTokens excludes cache (same native semantics as the direct Anthropic API), so a Claude session with prompt caching reports gen_ai.usage.input_tokens — and total_tokens — that differ by exactly cache_read + cache_write depending on which package instrumented it.

The adjudication question

The OTel GenAI conventions for Anthropic (semantic-conventions-genai, docs/gen-ai/anthropic.md) state cache read/write tokens MUST be added to native input tokens to compute gen_ai.usage.input_tokens. On that reading, the anthropic package follows the rule and the bedrock path (for Claude models) doesn't — but bedrock also serves non-Anthropic models where no such rule exists, which may be the reason for the split.

Which is the intended semantics for Claude-via-Bedrock? If the anthropic.md rule is meant to apply per-model rather than per-SDK-package, would aligning the bedrock Claude path preserve cross-entry-point comparability of token and cost figures? Happy to send a minimal repro (same system prompt + cache point, called via anthropic.messages.create and via bedrock Converse, asserting the input_tokens delta) or a small PR for the Claude-on-Bedrock branch.

Context: I maintain AgentMeasure, a measurement-conformance project; this is part of a public 30-projects/30-days check of agent-telemetry accounting semantics. Asking for the intended boundary, not claiming a defect.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Compare the Anthropic calculation in packages/opentelemetry-instrumentation-anthropic/.../anthropic/init.py and streaming.py with converse_usage_record in packages/opentelemetry-instrumentation-bedrock/.../bedrock/span_utils.py. Read docs/gen-ai/anthropic.md to establish the applicable convention, then reproduce the same cached Claude call through both entry points. Done means the intended Claude-via-Bedrock semantics are decided and any needed scope for alignment is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.