llm evals: break down evaluation results by event property
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Is your feature request related to a problem?
When viewing the results of an LLM evaluation, users can only see aggregate metrics (pass rate, runs count, failing count) across all matching $ai_generation events. There is no way to slice those results by an event property such as $ai_environment, a prompt version property, or any other custom dimension. This makes it hard to compare evaluation outcomes across environments (e.g. staging vs production) or across prompt variants in an A/B test, which are two common workflows for users iterating on LLM features.
Describe the solution you'd like
Add a "breakdown" option to the evaluation results view that takes any event property and renders a per-value breakdown of the existing summary cards, pass-rate trend chart, and runs table. A user looking at an evaluation should be able to pick $ai_environment (or any other property they capture) and see a side-by-side comparison of pass rate, run count, and failing count for each value of that property, without having to duplicate the evaluation. The breakdown should apply to both the trend chart and the summary, so users can compare environments at a glance and drill into the runs for each segment.
Describe alternatives you've considered
- Writing raw HogQL or Creating custom insights and dashboards against $ai_evaluation events. Gives full flexibility but bypasses the evaluation UX entirely. (currently the best workaround)
- Duplicating the evaluation and adding a property filter for each dimension. This is the current workaround. Works, but doubles LLM judge spend on overlapping generations, scales poorly when there are more than two values to compare, and leaves stale duplicates behind whenever the underlying eval prompt or config changes.
- Clicking the "Trend insight" button on the evaluation detail page, which opens a Trends insight pre-filtered to the evaluation. Native Trends breakdowns work there, but the user loses the summary cards, runs table, and AI patterns view, and the breakdown is not persisted with the evaluation.
Additional context
Customer feedback: would love to be able to break down by dimension when viewing LLM evaluation results. Two specific use cases called out:
- Running evals in staging (or any non-prod environment) and comparing pass rates against production before promoting a change.
- A/B testing prompts and comparing metrics between prompt variants.
Raised in: posthoghelp.zendesk.com/agent/tickets/58462 (moved to PostHog: https://us.posthog.com/project/2/support/tickets/59373)
Debug info
- PostHog Cloud, Debug information: [please copy/paste from https://us.posthog.com/settings/project-details#variables]
- PostHog Hobby self-hosted with
docker compose, version/commit: [please provide] - PostHog self-hosted with Kubernetes (deprecated, see
Sunsetting Kubernetes support), version/commit: [please provide]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the evaluation results view and the existing Trend insight flow referenced in the issue. Trace how summary cards, the pass-rate trend chart, runs table, and AI patterns view receive evaluation data. Done means users can select an event property and compare per-value summaries, trends, and runs without duplicating the evaluation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, react, typescript
- Domain
- ai, analytics, backend, frontend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100