A/B Test allocation logic may produce biased results
- Dominant language
- TypeScript
- Stars
- 45
- Forks
- 9
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 17
Description
The way users are currently allocated to test cohorts using their MVT ID may lead to biased test results if there is more than one A/B test running at a time. Please see [this doc](https://docs.google.com/document/d/16CKxPaPV3bxCGygCH9pxXK6OS8ResfCavC-lpo5uAHc/edit?usp=sharing) for further information.
One potential solution to this problem would be to allocate users to test cohorts based on a seeded value, similar to the way we do in [support-frontend](https://github.com/guardian/support-frontend/blob/e4258f9e00d46797edba98f1d7b3347ae5196031/support-frontend/assets/helpers/abTests/abtest.ts#L356-L359).
Contributor guide
Research direction
Start with the linked A/B testing document, then locate the current MVT ID cohort-allocation logic in csnx. Compare it with the seeded-value approach shown in support-frontend's abtest.ts example. Done means concurrent A/B tests no longer produce biased cohort results, with the behavior verified by relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- frontend, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100