lmcinnes / lmcinnes/umap

Saving UMAP model with more than 4096 samples results in crashing

Open
#874 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.3k
Forks
871
Avg merge
1d 13h
Merged PRs (30d)
5

Description

I am trying to save a UMAP model that got trained with more than 4096 samples and 420 features each. This results in crashing either the jupyter notebook or the editor I use. When I try to load the saved file I get an EOF Error probably because the file is corrupted since juypter notebook crashed before fully saving the file. This happens when using joblib or pickle. The versions I use are:

Python 3.10.2
umap-learn==0.5.3
numba==0.55.1
pynndescent==0.5.6

#Code
#UMAP instantiating and fitting, vectors is array of vectors
reducer = umap.UMAP(random_state=42, min_dist=0.0,n_neighbors=20).fit(vectors)
#saving via pickle
f_name = 'saving_example.sav'
pickle.dump(reducer, open(f_name, 'wb'))
#or saving via joblib
joblib.dump(reducer, f_name)

#in another file
#load model
#pickle
f_name = 'saving_example.sav'
loaded_model = pickle.load((open(f_name, 'rb')))
#joblib
loaded_reducer = joblib.load(f_name)

The crashing also happens when trying to transform new/different data to a model that was fitted using more than 4096 samples.
Can somebody help please?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with the shown UMAP fit using more than 4096 samples, then test the pickle.dump, joblib.dump, pickle.load, joblib.load, and transform entry points. Done means saving and loading the fitted model, and transforming new data, no longer crashes for this dataset size.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.