lmcinnes / lmcinnes/umap

Reproducibility Discrepancy in UMAP in Local Environment vs Deployed App

Open
#1,024 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8.3k
Forks
871
Avg merge
1d 13h
Merged PRs (30d)
5

Description

Hello,

I am writing to report a reproducibility issue encountered while dockerizing my app, which utilizes UMAP. Despite ensuring the use of the exact same dependencies' version and UMAP's hyperparameters, I observed different results between my Jupyter notebook and the deployed app on an AWS instance.

To illustrate the disparity, I have provided a screenshot:

![Screenshot 2023-06-19 093138](https://github.com/lmcinnes/umap/assets/56358730/f10df5be-35ed-4ffb-a921-5c86a8a463ab)

`umap_embeddings` represents the result obtained in the notebook, while `umap_embeddings_job` is the outcome when running the container on the AWS instance.

The code used in both environments is as follows:

```
docs = df['text'].str.lower().str.replace("\n", " ").values
embedding_model_sbert = SentenceTransformer(SENTENCE_TRANSFORMER_MODEL)
embeddings_sbert = embedding_model_sbert.encode(docs, show_progress_bar=True)

umap_model = UMAP(**UMAP_PARAMS)
umap_embeddings = umap_model.fit_transform(embeddings_sbert)
```
where the parameters used in UMAP are defined in this dictionary:
```
UMAP_PARAMS = dict(
n_neighbors=15,
n_components=5,
min_dist=0.0,
metric='cosine',
random_state=42
)
```

In both environments I am using Python version 3.8.10 and UMAP version 0.5.3. The list of UMAP's dependencies installed in both environments is as follows:
```
numpy==1.23.0
scipy==1.8.0
pynndescent==0.5.10
scikit-learn==1.0.2
```

Am I missing something?

Thank you in advance!

Contributor guide

Open the contributing guide

Research direction

No repository file or test is named. Start by comparing the notebook and AWS container at the UMAP fit_transform call, including the listed Python, UMAP, and dependency versions and the generated embeddings; done means identifying why the environments diverge and documenting or validating reproducible matching results.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, jupyter-notebook, numpy, python, scikit-learn
Domain
cloud, devops, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.