lmcinnes / lmcinnes/umap

Reproducibility depends on `n_neighbors`

Open
#892 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.3k
Forks
871
Avg merge
1d 13h
Merged PRs (30d)
5

Description

I encountered an issue with the reproducibility of UMAP v0.5.3 (installed via conda). Here's the code:

```python
import umap

import numpy as np

random_data = np.random.random((100, 1)).astype(np.float32)

for n_neighbors in range(2, 10):

mapper_1 = umap.UMAP(n_neighbors=n_neighbors, random_state=1337)
mapper_2 = umap.UMAP(n_neighbors=n_neighbors, random_state=1337)

embedding_1 = mapper_1.fit_transform(random_data)
embedding_2 = mapper_2.fit_transform(random_data)

distance = np.linalg.norm(embedding_1 - embedding_2)

print(f"{n_neighbors=}: {np.allclose(embedding_1, embedding_2)=}, {distance=}")

```

I get non-reproducible results for at least `n_neighbors=2` and `n_neighbors=3`. However, even this behavior is not reproducible:
```python
# output of first run of above code snippet
n_neighbors=2: np.allclose(embedding_1, embedding_2)=False, distance=107.652954
n_neighbors=3: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=4: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=5: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=6: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=7: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=8: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=9: np.allclose(embedding_1, embedding_2)=True, distance=0.0

# output of another run, i.e., with new random_data
n_neighbors=2: np.allclose(embedding_1, embedding_2)=False, distance=130.44937
n_neighbors=3: np.allclose(embedding_1, embedding_2)=False, distance=97.9669
n_neighbors=4: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=5: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=6: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=7: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=8: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=9: np.allclose(embedding_1, embedding_2)=True, distance=0.0
```

I'm glad for any help. Cheers! :slightly_smiling_face:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied Python reproducer with the reported UMAP v0.5.3 environment and compare repeated fit_transform results across n_neighbors values. Trace the reproducibility path for the affected neighbor counts; done means identical embeddings for repeated runs with the same random_state, or a documented explanation of the remaining limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.