PostHog / PostHog/posthog

Feature Request: Support experiments without a control variant (0% control use case)

Open
#34,832 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement experiments/new-engine team/experiments
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
7h 27m
Merged PRs (30d)
222

Description

Feature request

Is your feature request related to a problem?

Some teams run experiments iteratively and would like to compare multiple test variants without requiring a control group.

Right now, PostHog experiments require at least one variant to be marked as "control" and to have exposure events in order to calculate conversion metrics. If the control group is set to 0%, results will not show, even if test variants are receiving traffic.

Describe the solution you'd like

We’d like to run experiments with:

  • 0% control and 100% split across test variants
  • Full metric reporting and statistical significance between non-control variants
  • Ability to designate "control" as optional, or allow users to re-assign a previous variant as baseline without creating a new experiment

This would support workflows where:

  • A previously tested variant becomes the new default
  • Teams want to test improvements to that new baseline
  • Avoids having to reset the feature flag or re-implement tracking

Describe alternatives you've considered

  • Creating a new feature flag and starting a fresh experiment (adds implementation overhead)
  • Reusing the same flag but manually analyzing results with HogQL (non-trivial for many teams)
  • Temporarily assigning a “dummy” control variant at 1% (skews metrics and complicates interpretation)

Additional context

From: https://posthoghelp.zendesk.com/agent/tickets/33830 (moved to PostHog: https://us.posthog.com/project/2/support/tickets/35341)

Debug info
null

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by tracing how experiments require a control variant and how conversion metrics and statistical significance are calculated. Done means supporting a 0% control split, reporting metrics for test variants, and comparing non-control variants without requiring a new experiment.

Written by the indexing model from the issue text.

Assessment

Domain
analytics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.