PostHog / PostHog/posthog

LLM analytics: cost double-counts Gemini reasoning tokens (output tokens already inclusive)

Open
#71,065 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Bug

LLM analytics cost calculation prices Gemini reasoning tokens twice: $ai_output_tokens from the Vercel AI SDK already includes reasoning tokens, but the cost pipeline adds $ai_reasoning_tokens on top, inflating $ai_total_cost_usd.

Repro

gemini-2.5-flash via Vercel AI SDK v6, wrapped with @posthog/ai@8.3.0 withTracing (Node, posthog-node@5.36.2).

Provider usage returned by the SDK:

{"inputTokens":9,"outputTokens":22,"outputTokenDetails":{"textTokens":1,"reasoningTokens":21},"totalTokens":31,"raw":{"thoughtsTokenCount":21,"promptTokenCount":9,"candidatesTokenCount":1,"totalTokenCount":31}}

Event captured with $ai_input_tokens: 9, $ai_output_tokens: 22 (inclusive of the 21 reasoning tokens — matches Google's semantics, where candidatesTokenCount + thoughtsTokenCount are both billed as output), $ai_reasoning_tokens: 21.

Expected

At Gemini list prices ($0.30/M input, $2.50/M output incl. thinking):

9 × 0.30/1e6 + 22 × 2.50/1e6 = $0.0000577

Actual

$ai_total_cost_usd = 0.0001102

which is exactly 9 × 0.30/1e6 + (22 + 21) × 2.50/1e6 — the 21 reasoning tokens are priced a second time on top of the already-inclusive output count.

Control: a zero-reasoning call on the same stack (gemini-2.5-flash-lite, LangChain callback handler, 9 in / 1 out) prices exactly right at $0.0000013, isolating the discrepancy to reasoning-token handling.

Suggested fix

Same class of problem as inclusive/exclusive cache-token reporting, which already has auto-detection plus a $ai_cache_reporting_exclusive override. Reasoning tokens need the equivalent: auto-detect providers that report output inclusive of reasoning (Gemini / Vercel AI SDK), or support a $ai_reasoning_reporting_exclusive property.

Workaround we're using meanwhile: pre-calculated $ai_input_cost_usd / $ai_output_cost_usd via posthogProperties.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the LLM analytics cost pipeline and the Gemini/Vercel AI SDK usage mapping described in the reproduction. Compare how inclusive reasoning and output tokens are represented, then verify that the reported total cost prices the reasoning tokens only once while preserving the zero-reasoning control case.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js
Domain
ai, analytics
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.