umap crashes in my computer with 900,000 points
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.3k
- Forks
- 871
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 5
Description
Hi, I have been trying to embed 900,000 points using UMAP in my computer.
The program eventually gets killed by the system. I tried running in both Jupyter and in terminal.
My system: 16Core/32Thread AMD CPU, 128GB RAM (Terminal reports 125GB). Ubuntu 18.04.3 LTS.
I was wondering if it is a system requirement issue or an issue in how the UMAP handles this many points. (In the paper, it seems UMAP can handle millions of points as there is a visualization of 3Million points.)
Here is a code that reproduces the error in my computer:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
X_main = np.random.rand(900000, 1000)
n_components = 2
pca = PCA(n_components = 50)
X_train = pca.fit_transform(X_main)
n_neighbors= 50
MIN_DIST = 0.1
import umap
ump = umap.UMAP(n_neighbors=n_neighbors,
min_dist=MIN_DIST,
n_components=2,
random_state=100,
metric= 'euclidean')
y_umap = ump.fit_transform(X_train)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the supplied Python reproducer and compare memory and process behavior in terminal and Jupyter. Trace the UMAP fit_transform path for the 900,000-point, 50-component input and determine whether termination is an expected resource limit or a reproducible bug; no target file or test is named in the report.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter, matplotlib, numpy, python, scikit-learn
- Domain
- data-visualization, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100