PostHog / PostHog/ai-plugin

Two $ai_generation fidelity bugs in the Claude Code session ingest

Open
#92 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
84
Forks
15
Avg merge
14h 34m
Merged PRs (30d)
7

Description

Hi! Two small bugs in the Claude Code session-end ingest path that affect what PostHog sees on $ai_generation events. Both reproduce on the current main (a004aa2). Happy to PR if useful.

1. Cache token properties missing the $ai_ prefix

posthog_llma/events.pybuild_ai_generation emits:

properties = {
    ...
    "cache_read_input_tokens": cache_read_tokens,
    "cache_creation_input_tokens": cache_creation_tokens,
}

PostHog's LLM Analytics cost pipeline only reads the $ai_* namespace (docs call out $ai_cache_read_input_tokens / $ai_cache_creation_input_tokens), so cache reads/writes are dropped on the floor and $ai_total_cost_usd is computed from input+output only.

For Anthropic prompt-cached workloads the cache buckets dominate the bill, so reported cost can be ~25x lower than the actual spend. Concrete example from a real session: cache_read=150,109, cache_write=75,729, input=4, output=926 — PostHog showed ~$0.014, actual cost ~$0.343.

2. Extended thinking is emitted as {"type":"text"} instead of {"type":"thinking"}

posthog_llma/parser.py_finalize_generation concatenates thinking and text blocks into a single output_text string, and event_builder.pybuild_events wraps that as one text block:

content_blocks = []
if gen["output_text"]:
    content_blocks.append({"type": "text", "text": gen["output_text"]})

So when a Claude Code generation has both thinking and text blocks (common with extended thinking enabled), thinking content appears as plain assistant text in the PostHog UI instead of a dedicated thinking block — losing the visual distinction and breaking property queries that filter on block type. The streaming-merge work from #86 already tracks blocks by type in state["blocks_by_type"], so the typing info is right there in _finalize_generation, just collapsed before emission.

Thanks!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with posthog_llma/events.py build_ai_generation, then trace posthog_llma/parser.py finalize_generation and event_builder.py build_events. Use the blocks_by_type state described from #86 to understand the existing merge path. Done means cache token properties use the documented $ai namespace and extended-thinking output preserves separate thinking and text block types.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.