lmcinnes / lmcinnes/umap

umap crashes in my computer with 900,000 points

Open
#500 8 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.3k
Forks
871
Avg merge
1d 13h
Merged PRs (30d)
5

Description

Hi, I have been trying to embed 900,000 points using UMAP in my computer.
The program eventually gets killed by the system. I tried running in both Jupyter and in terminal.

My system: 16Core/32Thread AMD CPU, 128GB RAM (Terminal reports 125GB). Ubuntu 18.04.3 LTS.

I was wondering if it is a system requirement issue or an issue in how the UMAP handles this many points. (In the paper, it seems UMAP can handle millions of points as there is a visualization of 3Million points.)

Here is a code that reproduces the error in my computer:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA


X_main = np.random.rand(900000, 1000)

n_components = 2

pca = PCA(n_components = 50)
X_train = pca.fit_transform(X_main)


n_neighbors= 50
MIN_DIST = 0.1

import umap

ump = umap.UMAP(n_neighbors=n_neighbors,
        min_dist=MIN_DIST,
        n_components=2,
        random_state=100,
        metric= 'euclidean')

y_umap = ump.fit_transform(X_train)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied Python reproducer and compare memory and process behavior in terminal and Jupyter. Trace the UMAP fit_transform path for the 900,000-point, 50-component input and determine whether termination is an expected resource limit or a reproducible bug; no target file or test is named in the report.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter, matplotlib, numpy, python, scikit-learn
Domain
data-visualization, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.