Experiments AI Summary limitations
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Bug Description
Using the summarize feature on an experiment is hitting some regular limitations in response including
- I'm encountering the same technical issue with the experiment summary tool. Let me search for this experiment to get its details.
- ...but I encountered a technical issue when trying to generate an automated summary - it appears there may be a configuration problem with how the experiment's goal is set up.
- Unfortunately, I can't access the actual statistical results due to a technical issue with the experiment summary tool. To see the full results including variant performance and significance, I recommend viewing the experiment directly.
How to reproduce
- Navigate to an experiment
- click the Summarize button
Additional context
Reported by a customer and verified in a couple of other running experiments
The request would be to document any limitations and improve new capabilities for experiments here https://posthog.com/docs/posthog-ai/tools#limitations
Debug info
- [ ] PostHog Cloud, Debug information: [please copy/paste from https://us.posthog.com/settings/project-details#variables or https://eu.posthog.com/settings/project-details#variables]
- [ ] PostHog Hobby self-hosted with `docker compose`, version/commit: [please provide]
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the PostHog AI tools documentation and its limitations section linked in the issue. Compare the reported experiment-summary responses with the current documentation, then document confirmed limitations and clarify which requested capabilities remain unsupported. Done means the page accurately reflects the verified behavior and the reproduction steps are addressed.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100