fix(llma): trace summarization missing $ai_evaluation event type
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Bug
The trace summarization pipeline (posthog/temporal/llm_analytics/trace_summarization/) is missing $ai_evaluation from its AI event type lists:
constants.py—AI_EVENT_TYPEStuple is missing$ai_evaluationsampling.py— Two hardcodedWHERE event IN (...)SQL clauses are missing$ai_evaluation
This means any $ai_evaluation events within a trace are silently excluded from:
- Trace fetching (via
queries.pywhich uses theAI_EVENT_TYPESconstant) - Trace sampling (via the hardcoded SQL in
sampling.py)
The canonical Node.js list in nodejs/src/ingestion/ai/process-ai-event.ts includes $ai_evaluation, and the event type is used across the codebase (schema, taxonomy, usage reports, etc.).
Impact
Traces that contain evaluation events will have incomplete summaries — the evaluation results are dropped before the LLM summarizer sees them.
Expected behavior
$ai_evaluation should be included alongside the other AI event types in both the constants tuple and the sampling SQL queries.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in posthog/temporal/llm_analytics/trace_summarization/constants.py and sampling.py, then inspect queries.py to follow how the event lists are used. Add $ai_evaluation to the AI event tuple and both sampling SQL clauses, and verify that evaluation events are included in trace fetching and sampling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100