PostHog / PostHog/posthog

Auto-flag experiment metric changes (guardrail alerts + off-experiment anomalies)

Open
#67,202 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement team/experiments
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Feature request

Is your feature request related to a problem?

Experiment metrics compute direction, significance, and a metric goal but nothing watches them. A variant can be tanking a primary or guardrail metric and no one knows until they open the experiment.

Research surfaced two asks: (a) auto-flag when a metric moves too negatively against its goal, and (b) anomaly-detect significant changes in metrics not tied to the experiment ("did this experiment break something else?"). This is the clearest expression of the autonomy / instant-action thesis: learn from running experiments and act.

Describe the solution you'd like

  1. We can use the existing alerting system - AlertConfiguration along with the anomaly detectors. This could push a notification into PH, or a webhook for 3rd party platforms, or even push an alert into PHC and trigger a squence.
  2. We can make an AI anomaly-investigation agent/scout embedded in PHC - this would look on a scheduled basis, whilst an experiment is running.

Alert Types:

  1. Guardrail alerts - alert targets a metric or variant, and evaluates it against goal.
  2. Off-experiment anomaly watch - when an experiment runs, we watch over a set of watched org metrics and flag things that change within the experiment window.

Additional context

  • Solid experiment alerting system could begin with basic alerts to PHC when an experiment has completed, to begin auto-rollout, and suggest PR for experiment cleanup.
  • Strategically fits with our self-driving narrative.
Debug info
- [ ] PostHog Cloud, Debug information: [please copy/paste from https://us.posthog.com/settings/project-details#variables]
- [ ] PostHog Hobby self-hosted with `docker compose`, version/commit: [please provide]
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing AlertConfiguration and anomaly-detector implementations, then trace how experiment metrics expose direction, significance, and goals. Before coding, split the request into a scoped guardrail-alert milestone and a separate off-experiment anomaly watch; done should name supported alert targets, evaluation windows, notification paths, and tests for each.

Written by the indexing model from the issue text.

Assessment

Domain
analytics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.