lmcinnes / lmcinnes/umap

Random_state produces different results on different operating systems

Open
#153 9 comments 7 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.3k
Forks
871
Avg merge
1d 13h
Merged PRs (30d)
5

Description

### Issue

The `random_state` parameter produces deterministic results on a specific OS, but does not produce the same results on different OSes. Here are some examples for umap-learn, run with the following code. I used the example from the README here, as well as Scikit's check_random_state as a control (all Scikit results are the same).

The results are also seem to be dependent on the version of Numba that is installed.

```
# UMAP example with random state
import umap
from sklearn.datasets import load_digits

digits = load_digits()

embedding = umap.UMAP(
n_neighbors=5,
min_dist=0.3,
metric='correlation',
random_state=2018,
).fit_transform(digits.data)
embedding

# Scikit check random state
from sklearn.utils import check_random_state
random_state = check_random_state(2018)
random_state.rand(4)
```

### Example Results

Machine | Architecture | Python Version | umap-learn Version | numba Version | UMAP Result
-- | -- | -- | -- | -- | --
Macbook Pro #1 | Darwin C02P141DG3QD 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 21 20:07:39 PDT 2018; root:xnu-3789.73.14~1/RELEASE_X86_64 x86_64 | Python 3.7.0 | 0.3.2 | 0.39.0 | array([[16.42446  , -2.1266642],       [ 7.231049 , -1.5276358],       [-1.5864906, -5.1226635],       ...,       [ 6.094945 ,  1.2291753],       [ 1.3193432,  5.4169164],       [ 5.5729628,  2.2857437]], dtype=float32)
Macbook Pro #1 | Darwin C02P141DG3QD 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 21 20:07:39 PDT 2018; root:xnu-3789.73.14~1/RELEASE_X86_64 x86_64 | Python 3.7.0 | 0.3.5 | 0.40.1 | array([[32.471622,  8.842674],       [16.400652, 13.036578],       [ 9.181449,  3.948576],       ...,       [19.216055, 12.42009 ],       [ 6.522507, 14.285691],       [19.517092, 11.733169]], dtype=float32)
Macbook Pro #2 | Darwin C02VN4T7HV2L 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64 | Python 3.7.0 | 0.3.2 | 0.40.0 | array([[16.42446  , -2.1266642],        [ 7.231049 , -1.5276358],        [-1.5864906, -5.1226635],        ...,        [ 6.094945 ,  1.2291753],        [ 1.3193432,  5.4169164],        [ 5.5729628,  2.2857437]], dtype=float32)
Macbook Pro #2 | Darwin C02VN4T7HV2L 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64 | Python 3.7.0 | 0.3.5 | 0.40.1 | array([[32.471622,  8.842674],        [16.400652, 13.036578],        [ 9.181449,  3.948576],        ...,        [19.216055, 12.42009 ],        [ 6.522507, 14.285691],        [19.517092, 11.733169]], dtype=float32)
Debian Docker | Linux 389088ec7b25 4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018 x86_64 GNU/Linux | Python 3.5.3 | 0.3.5 | 0.40.1 | array([[25.864304 ,  7.870304 ],        [16.924606 ,  7.9489594],        [ 7.4818945,  9.081071 ],        ...,        [15.565144 , 10.721824 ],        [ 7.7764506, 14.354664 ],        [14.85415  , 11.515898 ]], dtype=float32)
Ubuntu Docker | Linux 6a9a07ef70b7 4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | Python 3.6.6 | 0.3.5 | 0.40.1 | array([[25.864304 ,  7.870304 ],        [16.924606 ,  7.9489594],        [ 7.4818945,  9.081071 ],        ...,        [15.565144 , 10.721824 ],        [ 7.7764506, 14.354664 ],        [14.85415  , 11.515898 ]], dtype=float32)
Ubuntu Docker | Linux 6a9a07ef70b7 4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | Python 3.7.0 | 0.3.5 | 0.40.1 | array([[25.864304 ,  7.870304 ],        [16.924606 ,  7.9489594],        [ 7.4818945,  9.081071 ],        ...,        [15.565144 , 10.721824 ],        [ 7.7764506, 14.354664 ],        [14.85415  , 11.515898 ]], dtype=float32)
Ubuntu Desktop | Linux brick 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | Python 3.6.6 | 0.3.5 | 0.40.1 | array([[25.864225 ,  7.8703256],        [16.92632  ,  7.943247 ],        [ 7.4819674,  9.081023 ],        ...,        [15.570685 , 10.72381  ],        [ 7.776701 , 14.354493 ],        [14.864248 , 11.530873 ]], dtype=float32)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the README UMAP example with random_state=2018 and compare it with sklearn's check_random_state control across the listed operating systems and versions. Investigate the UMAP and Numba version differences shown in the examples; done means the source of cross-OS and version-dependent results is identified and reproducibility expectations are established.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.