PostHog / PostHog/posthog

fix(llma): trace summarization missing $ai_evaluation event type

Open Beginner friendly
#53,532 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Bug

The trace summarization pipeline (posthog/temporal/llm_analytics/trace_summarization/) is missing $ai_evaluation from its AI event type lists:

  1. constants.pyAI_EVENT_TYPES tuple is missing $ai_evaluation
  2. sampling.py — Two hardcoded WHERE event IN (...) SQL clauses are missing $ai_evaluation

This means any $ai_evaluation events within a trace are silently excluded from:

  • Trace fetching (via queries.py which uses the AI_EVENT_TYPES constant)
  • Trace sampling (via the hardcoded SQL in sampling.py)

The canonical Node.js list in nodejs/src/ingestion/ai/process-ai-event.ts includes $ai_evaluation, and the event type is used across the codebase (schema, taxonomy, usage reports, etc.).

Impact

Traces that contain evaluation events will have incomplete summaries — the evaluation results are dropped before the LLM summarizer sees them.

Expected behavior

$ai_evaluation should be included alongside the other AI event types in both the constants tuple and the sampling SQL queries.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in posthog/temporal/llm_analytics/trace_summarization/constants.py and sampling.py, then inspect queries.py to follow how the event lists are used. Add $ai_evaluation to the AI event tuple and both sampling SQL clauses, and verify that evaluation events are included in trace fetching and sampling.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
ai, backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.