Reproducibility depends on `n_neighbors`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.3k
- Forks
- 871
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 5
Description
I encountered an issue with the reproducibility of UMAP v0.5.3 (installed via conda). Here's the code:
```python
import umap
import numpy as np
random_data = np.random.random((100, 1)).astype(np.float32)
for n_neighbors in range(2, 10):
mapper_1 = umap.UMAP(n_neighbors=n_neighbors, random_state=1337)
mapper_2 = umap.UMAP(n_neighbors=n_neighbors, random_state=1337)
embedding_1 = mapper_1.fit_transform(random_data)
embedding_2 = mapper_2.fit_transform(random_data)
distance = np.linalg.norm(embedding_1 - embedding_2)
print(f"{n_neighbors=}: {np.allclose(embedding_1, embedding_2)=}, {distance=}")
```
I get non-reproducible results for at least `n_neighbors=2` and `n_neighbors=3`. However, even this behavior is not reproducible:
```python
# output of first run of above code snippet
n_neighbors=2: np.allclose(embedding_1, embedding_2)=False, distance=107.652954
n_neighbors=3: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=4: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=5: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=6: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=7: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=8: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=9: np.allclose(embedding_1, embedding_2)=True, distance=0.0
# output of another run, i.e., with new random_data
n_neighbors=2: np.allclose(embedding_1, embedding_2)=False, distance=130.44937
n_neighbors=3: np.allclose(embedding_1, embedding_2)=False, distance=97.9669
n_neighbors=4: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=5: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=6: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=7: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=8: np.allclose(embedding_1, embedding_2)=True, distance=0.0
n_neighbors=9: np.allclose(embedding_1, embedding_2)=True, distance=0.0
```
I'm glad for any help. Cheers! :slightly_smiling_face:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the supplied Python reproducer with the reported UMAP v0.5.3 environment and compare repeated fit_transform results across n_neighbors values. Trace the reproducibility path for the affected neighbor counts; done means identical embeddings for repeated runs with the same random_state, or a documented explanation of the remaining limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100