lmcinnes / lmcinnes/umap

UMAP Resistance to Scale Changes, Data Drift

Open
#969 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.3k
Forks
871
Avg merge
1d 13h
Merged PRs (30d)
5

Description

Hi, apologies if this is obvious, but I wanted to ask how resistant UMAP was to differences in scale and offset. Ex, if our train and test datasets are slightly differently positioned(due to some slight data drift), how will that affect UMAP.

I conducted a simple experiment where I check UMAP's projections after drifting the data slightly in order to see how it is changed. I used sklearn's make_blobs function to create three clusters with 3 features(3d space). I then used UMAP to compress them into 2 components. I ran the transform function for both training and testing data as a baseline, and the results were expected. however, after shifting the test data by an offset, i found that, while there were still 3 clusters, the classes in each cluster were different:

No offset of test data:
image
UMAP projections:
image

Offset of 1(adding 1 to the test data):
image
UMAP projections:
image

Offset of 2(adding 2 to the test data):
image
UMAP Projections:
image

While the relative distances between classes are the same, and there are still 3 clusters in similar positions, it seems like the data in each cluster changes. This is a little surprising to me and I thought that UMAP would be a little more resistant to scale changes(keep in mind though that i am only showing the first two features when there are three, however the point still stands) -- doesn't umap utilize graph-based methods to learn the manifold? Shouldn't it pay attention to the distribution of the test data rather than making static thresholds? Let me know if i am understanding this correctly.

The use-case for this is during cases with slight train-test drift(ex data is highly variable and while the distribution of clusters is similar it can shift from time to time)

Here is a colab: https://colab.research.google.com/drive/1NyBNSxa81skCMDkOGfpVeHt5Nj3awkjH?usp=sharing

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked Colab and reproduce the make_blobs experiment using UMAP's fit and transform calls for the baseline and shifted test data. Compare how the offset changes projected cluster assignments, then document whether the observed behavior is expected and what scale or drift handling users should apply.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.