guardian / guardian/csnx

A/B Test allocation logic may produce biased results

Open
#384 2 comments 0 reactions 0 assignees View on GitHub
Bug enhancement
Dominant language
TypeScript
Stars
45
Forks
9
Avg merge
1d 9h
Merged PRs (30d)
17

Description

The way users are currently allocated to test cohorts using their MVT ID may lead to biased test results if there is more than one A/B test running at a time. Please see [this doc](https://docs.google.com/document/d/16CKxPaPV3bxCGygCH9pxXK6OS8ResfCavC-lpo5uAHc/edit?usp=sharing) for further information.

One potential solution to this problem would be to allocate users to test cohorts based on a seeded value, similar to the way we do in [support-frontend](https://github.com/guardian/support-frontend/blob/e4258f9e00d46797edba98f1d7b3347ae5196031/support-frontend/assets/helpers/abTests/abtest.ts#L356-L359).

Contributor guide

Open the contributing guide

Research direction

Start with the linked A/B testing document, then locate the current MVT ID cohort-allocation logic in csnx. Compare it with the seeded-value approach shown in support-frontend's abtest.ts example. Done means concurrent A/B tests no longer produce biased cohort results, with the behavior verified by relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
frontend, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.