Feature Request: Create a warning when experiments have high a "$multiple"
Open
Nobody has claimed this yet.
enhancement
team/experiments
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Feature request
Is your feature request related to a problem?
- Experiments can become contaminated if too many users users have multiple variant exposures.
- Users don't always know is considered a safe threshold for multiple exposures and what would be causing data problems.
Describe the solution you'd like
- Right now we surface SRM errors.
- It would be great if we could do the same for a high
$multipleexposure count - Perhaps users can set a
Describe alternatives you've considered
- This information is displayed in the UI, but it can be a bit hidden since the exposures tab is default hidden.
- So you can see it, but just have to figure out what a proper variance is.
Additional context
- From https://posthoghelp.zendesk.com/agent/tickets/58505 (moved to PostHog: https://us.posthog.com/project/2/support/tickets/60308)
Debug info
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the linked SRM guidance and the existing $multiple information in the exposures tab. Clarify the safe threshold, whether users can configure it, and the warning’s explanation; done means high multiple exposure is surfaced with actionable guidance without requiring the exposures tab to be opened.
Written by the indexing model from the issue text.
Assessment
- Domain
- analytics
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100