Experiments: split metric results by metric event property (effect decomposition, full exposure denominator)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Feature request
Allow users to easily split a metric's results by a metric event property, producing one result series per property value — effectively a fast way to add N metrics, one per value.
Background
This came out of a customer (Arena) question about experiment breakdowns. Today, the experiment breakdown feature does segment analysis: for each value of a property/dimension, it recomputes the experiment result using only users in that segment, attributing each user to a segment via the value on their exposure event. The denominator (sample size) therefore adjusts to only the users in that subgroup.
That's correct and matches what other vendors do, but it's the wrong tool when a property can take multiple values per user within a session (e.g. a "mode" the user switches between). The treatment can shift the distribution of traffic across those values, so attributing by the first/exposure value looks imbalanced.
What's actually wanted: effect decomposition
Instead of analyzing the metric per audience segment, the user wants to split the metric itself into N metrics, one per value of a metric-event property — where the denominator stays the full exposure count. This is a distinct feature from the existing breakdown (segment analysis).
Proposed
Let users split a metric's results based on a metric event property, with each split sharing the full exposure denominator rather than a per-segment denominator.
Workaround today
Create a set of Shared metrics (one per property value) and use tags to add them all to an experiment at once.
Filed from a customer conversation — see internal Slack thread for full context.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing experiment breakdown behavior and the Shared metrics workaround described in the issue, then trace where metric-event properties and experiment denominators are handled. Done means users can split a metric into one result series per metric-event property value while every series retains the full exposure denominator.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- analytics
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100