PostHog / PostHog/posthog

Experiments AI Summary limitations

Open
#46,757 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

AI bug feature/posthog-ai team/experiments
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Bug Description

Using the summarize feature on an experiment is hitting some regular limitations in response including

  • I'm encountering the same technical issue with the experiment summary tool. Let me search for this experiment to get its details.
  • ...but I encountered a technical issue when trying to generate an automated summary - it appears there may be a configuration problem with how the experiment's goal is set up.
  • Unfortunately, I can't access the actual statistical results due to a technical issue with the experiment summary tool. To see the full results including variant performance and significance, I recommend viewing the experiment directly.

How to reproduce

  1. Navigate to an experiment
  2. click the Summarize button

Additional context

Reported by a customer and verified in a couple of other running experiments

The request would be to document any limitations and improve new capabilities for experiments here https://posthog.com/docs/posthog-ai/tools#limitations

Debug info
- [ ] PostHog Cloud, Debug information: [please copy/paste from https://us.posthog.com/settings/project-details#variables or https://eu.posthog.com/settings/project-details#variables]
- [ ] PostHog Hobby self-hosted with `docker compose`, version/commit: [please provide]
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the PostHog AI tools documentation and its limitations section linked in the issue. Compare the reported experiment-summary responses with the current documentation, then document confirmed limitations and clarify which requested capabilities remain unsupported. Done means the page accurately reflects the verified behavior and the reproduction steps are addressed.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.