traceloop / traceloop/openllmetry
Same cached Claude call via anthropic vs bedrock packages emits different gen_ai.usage.input_tokens — which is intended?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.4k
- Forks
- 1.1k
- Avg merge
- 8d 14h
- Merged PRs (30d)
- 2
Description
A cross-package semantics question found while auditing cache-token accounting against the OTel GenAI conventions.
The observation
For the same cached Claude call, two instrumentation packages in this repo emit different gen_ai.usage.input_tokens:
- anthropic package folds cache in:
packages/opentelemetry-instrumentation-anthropic/.../anthropic/__init__.py(~L226-234):input_tokens = prompt_tokens + cache_read_tokens + cache_creation_tokens, with cache subset attributes emitted alongside (streaming path matches,streaming.py~L85-105). - bedrock package does not:
packages/opentelemetry-instrumentation-bedrock/.../bedrock/span_utils.py(converse_usage_record, ~L1071-1099):input_tokens = usage.inputTokensas-is, withcacheReadInputTokens/cacheWriteInputTokensemitted only as separate attributes.
For Claude-on-Bedrock via Converse, the provider's inputTokens excludes cache (same native semantics as the direct Anthropic API), so a Claude session with prompt caching reports gen_ai.usage.input_tokens — and total_tokens — that differ by exactly cache_read + cache_write depending on which package instrumented it.
The adjudication question
The OTel GenAI conventions for Anthropic (semantic-conventions-genai, docs/gen-ai/anthropic.md) state cache read/write tokens MUST be added to native input tokens to compute gen_ai.usage.input_tokens. On that reading, the anthropic package follows the rule and the bedrock path (for Claude models) doesn't — but bedrock also serves non-Anthropic models where no such rule exists, which may be the reason for the split.
Which is the intended semantics for Claude-via-Bedrock? If the anthropic.md rule is meant to apply per-model rather than per-SDK-package, would aligning the bedrock Claude path preserve cross-entry-point comparability of token and cost figures? Happy to send a minimal repro (same system prompt + cache point, called via anthropic.messages.create and via bedrock Converse, asserting the input_tokens delta) or a small PR for the Claude-on-Bedrock branch.
Context: I maintain AgentMeasure, a measurement-conformance project; this is part of a public 30-projects/30-days check of agent-telemetry accounting semantics. Asking for the intended boundary, not claiming a defect.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Compare the Anthropic calculation in packages/opentelemetry-instrumentation-anthropic/.../anthropic/init.py and streaming.py with converse_usage_record in packages/opentelemetry-instrumentation-bedrock/.../bedrock/span_utils.py. Read docs/gen-ai/anthropic.md to establish the applicable convention, then reproduce the same cached Claude call through both entry points. Done means the intended Claude-via-Bedrock semantics are decided and any needed scope for alignment is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100