PolicyEngine / PolicyEngine/policybench

Record Anthropic thinking tokens from the raw response (litellm reports 0)

Open
#140 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2
Forks
3
Avg merge
12h 7m
Merged PRs (30d)
13

Description

litellm 1.88.1 estimates Anthropic reasoning tokens by token-counting the visible thinking text instead of reading the API's usage.output_tokens_details.thinking_tokens; Claude Opus 5 / Fable 5 / Sonnet 5 default to display: "omitted" (empty thinking text), so completion_tokens_details.reasoning_tokens is structurally 0 for them. Filed upstream with a proposed fix: BerriAI/litellm#36290.

Until that lands, our per-row usage recording (and the published usage_summary.csv) shows zero reasoning tokens for every Claude row regardless of actual thinking. Options:

  • Prefer the raw provider usage when available: read output_tokens_details.thinking_tokens off the response and record it in our reasoning column directly, falling back to litellm's field for other providers.
  • Regenerate the affected analysis CSVs for runs where thinking actually occurred (the tool_choice-auto sensitivity run is the first).

The board's canonical runs are unaffected in substance (Claude genuinely produced no thinking under the forced tool — see sensitivity/claude-thinking-2026-08.md), but the column should stop being wrong-by-construction before any v2 run where Claude thinking engages (#139).

🤖 Generated with Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the per-row usage recording and the code that writes reasoning tokens to usage_summary.csv, then inspect how the raw response usage is represented. Verify the change against the affected analysis CSVs, especially the tool_choice-auto sensitivity run, and confirm that provider responses with thinking report the expected reasoning count while other providers retain their existing fallback.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.