lmcinnes / lmcinnes/umap

Pickling fitted UMAP on sparse array with more than 4096 rows leads to an error

Open
#674 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.3k
Forks
871
Avg merge
1d 13h
Merged PRs (30d)
5

Description

I found a couple of similar issues #556 #547, but none exactly like the one I describe I believe. Apologies if this has been already covered in one of the other issues.

Here is the minimal reproducible example:

```python
import pickle
import scipy
import umap

mat = scipy.sparse.random(4097, 100)

reducer = umap.UMAP()

reducer.fit_transform(mat)

with open('test.pkl', 'wb') as f:
pickle.dump(reducer, f)

```

It returns:

```bash
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of type(CPUDispatcher()) with parameters (readonly array(float32, 2d, C), float32, readonly array(float32, 1d, C), array(int64, 1d, C))
Known signatures:
* (array(float32, 1d, C), float32, array(float32, 1d, C), array(int64, 1d, C)) -> bool
* (readonly array(float32, 1d, C), float32, readonly array(float32, 1d, C), array(int64, 1d, C)) -> bool
During: resolving callee type: type(CPUDispatcher())
During: typing of call at /Users/campea/notebooks_experimental/env/lib/python3.8/site-packages/pynndescent/pynndescent_.py (1181)

File "env/lib/python3.8/site-packages/pynndescent/pynndescent_.py", line 1181:
def tree_search_closure(point, rng_state):

while tree_children[node, 0] > 0:
side = select_side(
^
```

Note that if we replace `reducer.fit_transform(mat)` for `reducer.fit_transform(mat.todense())` it works, but that means I have to kill the sparse array structure, which is not ideal.

I reproduced the error on a minimal virtualenv (python3.8) with only UMAP and its deps:

```bash
joblib==1.0.1
llvmlite==0.36.0
numba==0.53.1
numpy==1.20.3
pynndescent==0.5.2
scikit-learn==0.24.2
scipy==1.6.3
threadpoolctl==2.1.0
umap-learn==0.5.1
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the minimal Python reproduction using a 4097-row scipy sparse matrix and the traceback at pynndescent_.py line 1181. Compare sparse and dense fit_transform paths, then verify that fitting and pickling the sparse-input reducer succeeds without the reported numba error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.