Auto-flag experiment metric changes (guardrail alerts + off-experiment anomalies)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Feature request
Is your feature request related to a problem?
Experiment metrics compute direction, significance, and a metric goal but nothing watches them. A variant can be tanking a primary or guardrail metric and no one knows until they open the experiment.
Research surfaced two asks: (a) auto-flag when a metric moves too negatively against its goal, and (b) anomaly-detect significant changes in metrics not tied to the experiment ("did this experiment break something else?"). This is the clearest expression of the autonomy / instant-action thesis: learn from running experiments and act.
Describe the solution you'd like
- We can use the existing alerting system -
AlertConfigurationalong with the anomaly detectors. This could push a notification into PH, or a webhook for 3rd party platforms, or even push an alert into PHC and trigger a squence. - We can make an AI anomaly-investigation agent/scout embedded in PHC - this would look on a scheduled basis, whilst an experiment is running.
Alert Types:
- Guardrail alerts - alert targets a metric or variant, and evaluates it against goal.
- Off-experiment anomaly watch - when an experiment runs, we watch over a set of watched org metrics and flag things that change within the experiment window.
Additional context
- Solid experiment alerting system could begin with basic alerts to PHC when an experiment has completed, to begin auto-rollout, and suggest PR for experiment cleanup.
- Strategically fits with our self-driving narrative.
Debug info
- [ ] PostHog Cloud, Debug information: [please copy/paste from https://us.posthog.com/settings/project-details#variables]
- [ ] PostHog Hobby self-hosted with `docker compose`, version/commit: [please provide]
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing AlertConfiguration and anomaly-detector implementations, then trace how experiment metrics expose direction, significance, and goals. Before coding, split the request into a scoped guardrail-alert milestone and a separate off-experiment anomaly watch; done should name supported alert targets, evaluation windows, notification paths, and tests for each.
Written by the indexing model from the issue text.
Assessment
- Domain
- analytics
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100