humanmade / humanmade/aws-analytics

Make A/B test queries only track unique users & conversions

Open
#314 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
19
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Right now the aggregated stats collected for A/B tests every hour are for any unique page session rather than unique visitor. This isn't really the industry standard and is a limitation of sampling and aggregating data over time.

We could sample by unique visitors only but it would be unique visitors for an hour at a time, there might be cross-over between hourly sampling.

Needs some research to understand / how if this can be done. Querying up to 90 days worth of data, although an option, could be very slow indeed.

We could possibly exclude users once they've been exposed to a test once, but that would exclude them if they were to convert at a later date.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.