traceloop / traceloop/openllmetry

anthropic: reasoning-token subset not emitted — cross-provider reasoning cost invisible

Open
#4,458 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.4k
Forks
1.1k
Avg merge
8d 14h
Merged PRs (30d)
2

Description

A cross-project finding from the 30-project conformance check — relevant to OpenLLMetry after #4449 (cross-package cache divergence).

The observation

The anthropic instrumentation package emits the reasoning-token subset (gen_ai.usage.details.reasoning_tokens or equivalent for OpenAI), but the Anthropic direct path does not emit Claude's extended-thinking reasoning tokens at all:

  • packages/opentelemetry-instrumentation-anthropic/.../anthropic/__init__.py and streaming.py: only input/output/cache_read/cache_write/total are extracted
  • usage.output_tokens_details.reasoning_tokens (present in Anthropic's API response for Claude with extended thinking) is not read
  • No gen_ai.usage.details.reasoning_tokens or similar attribute is emitted for Anthropic models

Meanwhile:

  • The OpenAI package (chat_wrappers.py ~L334-353, responses_wrappers.py ~L444-451) correctly emits reasoning_tokens from completion_tokens_details / output_tokens_details
  • The Google Vertex AI package (span_utils.py ~L293-322) correctly emits cached_content_token_count (though thoughts_token_count is also missing)

The consequence

"Output tokens" is semantically different across providers in the emitted telemetry:

  • OpenAI: output_tokens includes reasoning; reasoning subset is separately visible
  • Anthropic: output_tokens includes thinking; reasoning subset is NOT separately visible
  • Google: output_tokens includes thoughts (sometimes); thoughts subset is NOT visible

For any consumer computing reasoning-token cost separately (reasoning is priced differently on some models), Anthropic's thinking tokens are invisible. For consumers comparing reasoning ratios across providers, the data simply doesn't exist for Anthropic.

The question

Would it make sense to emit reasoning_tokens from usage.output_tokens_details on the Anthropic path, matching what the OpenAI path already does? The field exists in the API response when Claude uses extended thinking.

Context: AgentMeasure conformance check (30 projects / 30 days); this is the OpenLLMetry follow-up after #4449. The Anthropic cache normalization (which we raised) is the most carefully implemented we've audited — this is the one remaining gap.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in packages/opentelemetry-instrumentation-anthropic/.../anthropic/init.py and streaming.py, then compare the reasoning-token handling in OpenAI's chat_wrappers.py and responses_wrappers.py. Confirm how usage.output_tokens_details.reasoning_tokens appears in Anthropic responses and ensure the direct and streaming paths expose the reasoning-token subset consistently; done means Anthropic reasoning tokens are visible to telemetry consumers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.