PostHog / PostHog/posthog

Experiments: split metric results by metric event property (effect decomposition, full exposure denominator)

Open
#64,075 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
39.9k
Forks
3.4k
Avg merge
6h 51m
Merged PRs (30d)
232

Description

Feature request

Allow users to easily split a metric's results by a metric event property, producing one result series per property value — effectively a fast way to add N metrics, one per value.

Background

This came out of a customer (Arena) question about experiment breakdowns. Today, the experiment breakdown feature does segment analysis: for each value of a property/dimension, it recomputes the experiment result using only users in that segment, attributing each user to a segment via the value on their exposure event. The denominator (sample size) therefore adjusts to only the users in that subgroup.

That's correct and matches what other vendors do, but it's the wrong tool when a property can take multiple values per user within a session (e.g. a "mode" the user switches between). The treatment can shift the distribution of traffic across those values, so attributing by the first/exposure value looks imbalanced.

What's actually wanted: effect decomposition

Instead of analyzing the metric per audience segment, the user wants to split the metric itself into N metrics, one per value of a metric-event property — where the denominator stays the full exposure count. This is a distinct feature from the existing breakdown (segment analysis).

Proposed

Let users split a metric's results based on a metric event property, with each split sharing the full exposure denominator rather than a per-segment denominator.

Workaround today

Create a set of Shared metrics (one per property value) and use tags to add them all to an experiment at once.


Filed from a customer conversation — see internal Slack thread for full context.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing experiment breakdown behavior and the Shared metrics workaround described in the issue, then trace where metric-event properties and experiment denominators are handled. Done means users can split a metric into one result series per metric-event property value while every series retains the full exposure denominator.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
analytics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.