MetaMask / MetaMask/metamask-mobile
Enable A/B Testing in RemoteFeatureFlag
- Dominant language
- TypeScript
- Stars
- 3k
- Forks
- 1.7k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 669
Description
### What is this about?
This ticket is mean to support A/B testing within the remote feature flag controller. It will require logic to use the `metaMetricsID` and `profileID`. The `metaMetricsID` will be available when the app starts and the `profileID` will be available after a wallet is created.
This will require logic to handle the use of each of these IDs.
# 1 Use Profile ID for A/B tests and use MetametricsID as a fallback
- Profile ID gives us deduplication, makes a/b much more consistent.
- Use profile ID of a primary account
- During the onboarding, when profile_id is not yet available - use metametrics_id instead
The goal is to be able to run an A/B test on a new user's experience, e.g. on the first time the user ever sees the Metamask home page
To clarify, when the new user is onboarding, what happens first:
- Profile ID becomes available and can be used
- Feature Flag controller is initialized
Based on this, define if we can always use profile_id fo a/b tests or we have to have a fallback to metametricsId (or a replacement installation identifier)
When checking for flags:
- If metametrics_id is used AND the a/b test does not have a tag “new_user” - always return false
- If metametrics_id is used AND the a/b test has a tag “new_user” - evaluate as usual
- If profile_id is used AND the a/b test does not have a tag “new_user” - evaluate as usual
- If profile_id is used AND the a/b test has a tag “new_user” - always return false
# 2 Make a/b tests independent
- Right now, only the ID is used as a seed to determine what bucket the user falls into in an a/b test
- This means that, if there’s more than 1 a/b test configured - they will never be independent of each other
- To solve this, we need to:
- Define the distribution using profile/metametrics ID + a/b test ID as a seed instead of just profile/metametrics ID
### Scenario
_No response_
### Design
_No response_
### Technical Details
- Requires changes to [remote-feature-flag-controller](https://github.com/MetaMask/core/blob/0379cb76e5d22212fd963fea9f8877f845ac3b17/packages/remote-feature-flag-controller/src/remote-feature-flag-controller.ts#L131C1-L132C1)
- Requires updates to client code when profileID becomes available
How A/B testing would work
- MetaMask profile ID should already be generated (with the exception on onboarding, where we'll fallback to use MetaMetrics ID)
- We'll need to pass the ID to the RemoteFeatureFlagController config
- RemoteFeatureFlagController will then fetch feature flags from MetaMask server
- MetaMask server returns entire list of feature flags to the controller
- Payload will have some A/B testing flags, where the data will include scope w/ thresholds
- RemoteFeatureFlagController will perform numeric derivation from combination of (Profile ID or Metametrics ID + feature flag key) to generate a number (0 > N < 1)
- RemoteFeatureFlagController will compare and update A/B testing flags based on the number generated to determine which variant to serve to the the user
#### Summary
1. MetaMask Profile ID Generation & Fallback
Create helper to retrieve profile ID from AuthenticationController
Fallback to MetaMetrics ID during onboarding
Handle async operations and error cases
2. Pass ID to RemoteFeatureFlagController Config
Extend controller initialization to accept profile ID getter or update the metametricsId input to identity input
Require package updates to support new config option
Ensure reactive updates when profile ID becomes available
3. RemoteFeatureFlagController Fetches Feature Flags
this is already completed no additional work is required on this
4. MetaMask Server Returns A/B Testing Data
this is already completed no additional work is required on this
5. Numeric Derivation from Profile ID + Flag Key
There is currently a deterministic function called generateDeterministicRandomNumber. use a combination of the flag key and profileId/metametrics as input to determine new number. It should covert the input to number between 0-1.
include this change in the package update.
Add tests to ensure consistency.
6. Compare and Update A/B Testing Flags
This is already implemented but the input from the generateDeterministicRandomNumber may change the resulting category the user is placed in.
7. State Management and UI Updates
No additional work is needed here. Just ensure the updates are correct.
### Threat Modeling Framework
_No response_
### Acceptance Criteria
_No response_
### Stakeholder review needed before the work gets merged
- [ ] Engineering (needed in most cases)
- [ ] Design
- [ ] Product
- [ ] QA (automation tests are required to pass before merging PRs but not all changes are covered by automation tests - please review if QA is needed beyond automation tests)
- [ ] Security
- [ ] Legal
- [ ] Marketing
- [ ] Management (please specify)
- [ ] Other (please specify)
### References
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.