UMAP Resistance to Scale Changes, Data Drift
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.3k
- Forks
- 871
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 5
Description
Hi, apologies if this is obvious, but I wanted to ask how resistant UMAP was to differences in scale and offset. Ex, if our train and test datasets are slightly differently positioned(due to some slight data drift), how will that affect UMAP.
I conducted a simple experiment where I check UMAP's projections after drifting the data slightly in order to see how it is changed. I used sklearn's make_blobs function to create three clusters with 3 features(3d space). I then used UMAP to compress them into 2 components. I ran the transform function for both training and testing data as a baseline, and the results were expected. however, after shifting the test data by an offset, i found that, while there were still 3 clusters, the classes in each cluster were different:
No offset of test data:

UMAP projections:

Offset of 1(adding 1 to the test data):

UMAP projections:

Offset of 2(adding 2 to the test data):

UMAP Projections:

While the relative distances between classes are the same, and there are still 3 clusters in similar positions, it seems like the data in each cluster changes. This is a little surprising to me and I thought that UMAP would be a little more resistant to scale changes(keep in mind though that i am only showing the first two features when there are three, however the point still stands) -- doesn't umap utilize graph-based methods to learn the manifold? Shouldn't it pay attention to the distribution of the test data rather than making static thresholds? Let me know if i am understanding this correctly.
The use-case for this is during cases with slight train-test drift(ex data is highly variable and while the distribution of clusters is similar it can shift from time to time)
Here is a colab: https://colab.research.google.com/drive/1NyBNSxa81skCMDkOGfpVeHt5Nj3awkjH?usp=sharing
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked Colab and reproduce the make_blobs experiment using UMAP's fit and transform calls for the baseline and shifted test data. Compare how the offset changes projected cluster assignments, then document whether the observed behavior is expected and what scale or drift handling users should apply.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100