traceloop / traceloop/openllmetry
anthropic: reasoning-token subset not emitted — cross-provider reasoning cost invisible
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.4k
- Forks
- 1.1k
- Avg merge
- 8d 14h
- Merged PRs (30d)
- 2
Description
A cross-project finding from the 30-project conformance check — relevant to OpenLLMetry after #4449 (cross-package cache divergence).
The observation
The anthropic instrumentation package emits the reasoning-token subset (gen_ai.usage.details.reasoning_tokens or equivalent for OpenAI), but the Anthropic direct path does not emit Claude's extended-thinking reasoning tokens at all:
packages/opentelemetry-instrumentation-anthropic/.../anthropic/__init__.pyandstreaming.py: only input/output/cache_read/cache_write/total are extractedusage.output_tokens_details.reasoning_tokens(present in Anthropic's API response for Claude with extended thinking) is not read- No
gen_ai.usage.details.reasoning_tokensor similar attribute is emitted for Anthropic models
Meanwhile:
- The OpenAI package (
chat_wrappers.py~L334-353,responses_wrappers.py~L444-451) correctly emitsreasoning_tokensfromcompletion_tokens_details/output_tokens_details - The Google Vertex AI package (
span_utils.py~L293-322) correctly emitscached_content_token_count(thoughthoughts_token_countis also missing)
The consequence
"Output tokens" is semantically different across providers in the emitted telemetry:
- OpenAI: output_tokens includes reasoning; reasoning subset is separately visible
- Anthropic: output_tokens includes thinking; reasoning subset is NOT separately visible
- Google: output_tokens includes thoughts (sometimes); thoughts subset is NOT visible
For any consumer computing reasoning-token cost separately (reasoning is priced differently on some models), Anthropic's thinking tokens are invisible. For consumers comparing reasoning ratios across providers, the data simply doesn't exist for Anthropic.
The question
Would it make sense to emit reasoning_tokens from usage.output_tokens_details on the Anthropic path, matching what the OpenAI path already does? The field exists in the API response when Claude uses extended thinking.
Context: AgentMeasure conformance check (30 projects / 30 days); this is the OpenLLMetry follow-up after #4449. The Anthropic cache normalization (which we raised) is the most carefully implemented we've audited — this is the one remaining gap.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in packages/opentelemetry-instrumentation-anthropic/.../anthropic/init.py and streaming.py, then compare the reasoning-token handling in OpenAI's chat_wrappers.py and responses_wrappers.py. Confirm how usage.output_tokens_details.reasoning_tokens appears in Anthropic responses and ensure the direct and streaming paths expose the reasoning-token subset consistently; done means Anthropic reasoning tokens are visible to telemetry consumers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, observability
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100