PostHog / PostHog/posthog

LLM analytics trace totalCost excludes $ai_span $ai_total_cost_usd

Open
#47,423 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Summary

LLM Analytics trace-level totalCost currently appears to exclude $ai_total_cost_usd from $ai_span events.

In our traces, custom operations (for example reranking) are captured as $ai_span with valid $ai_total_cost_usd. The span-level UI shows the cost, but the trace header/table totalCost does not include it.

Expected behavior

Trace/session totalCost should include all AI event costs that belong to the trace, including $ai_span events with $ai_total_cost_usd.

Actual behavior

Trace/session totalCost includes only $ai_generation and $ai_embedding costs.

Evidence in code

Current trace queries sum cost only for event IN ('$ai_generation', '$ai_embedding'):

  • posthog/hogql_queries/ai/trace_query_runner.py
  • posthog/hogql_queries/ai/traces_query_runner.py

Both queries compute total_cost using:

sumIf(toFloat(properties.$ai_total_cost_usd), event IN ('$ai_generation', '$ai_embedding'))

Repro

  1. Send one $ai_generation event with $ai_total_cost_usd = 0.058
  2. Send one $ai_span event in the same trace with $ai_total_cost_usd = 0.004
  3. Open trace in LLM Analytics

Result:

  • Span shows 0.004 correctly
  • Trace totalCost shows 0.058 instead of 0.062

Notes

This makes trace/session totals under-report costs whenever teams put billable work on $ai_span (rerank/search/tool/API operations) instead of $ai_generation/$ai_embedding.

If this is intentional, could docs explicitly state that trace/session totals only aggregate generation/embedding events?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with posthog/hogql_queries/ai/trace_query_runner.py and posthog/hogql_queries/ai/traces_query_runner.py, focusing on the total_cost expressions and their event filters. Verify the trace and session totals against the repro values, including the $ai_span cost, and ensure both queries report the expected aggregate.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
ai, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.