Pickling fitted UMAP on sparse array with more than 4096 rows leads to an error
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.3k
- Forks
- 871
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 5
Description
I found a couple of similar issues #556 #547, but none exactly like the one I describe I believe. Apologies if this has been already covered in one of the other issues.
Here is the minimal reproducible example:
```python
import pickle
import scipy
import umap
mat = scipy.sparse.random(4097, 100)
reducer = umap.UMAP()
reducer.fit_transform(mat)
with open('test.pkl', 'wb') as f:
pickle.dump(reducer, f)
```
It returns:
```bash
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Invalid use of type(CPUDispatcher()) with parameters (readonly array(float32, 2d, C), float32, readonly array(float32, 1d, C), array(int64, 1d, C))
Known signatures:
* (array(float32, 1d, C), float32, array(float32, 1d, C), array(int64, 1d, C)) -> bool
* (readonly array(float32, 1d, C), float32, readonly array(float32, 1d, C), array(int64, 1d, C)) -> bool
During: resolving callee type: type(CPUDispatcher())
During: typing of call at /Users/campea/notebooks_experimental/env/lib/python3.8/site-packages/pynndescent/pynndescent_.py (1181)
File "env/lib/python3.8/site-packages/pynndescent/pynndescent_.py", line 1181:
def tree_search_closure(point, rng_state):
while tree_children[node, 0] > 0:
side = select_side(
^
```
Note that if we replace `reducer.fit_transform(mat)` for `reducer.fit_transform(mat.todense())` it works, but that means I have to kill the sparse array structure, which is not ideal.
I reproduced the error on a minimal virtualenv (python3.8) with only UMAP and its deps:
```bash
joblib==1.0.1
llvmlite==0.36.0
numba==0.53.1
numpy==1.20.3
pynndescent==0.5.2
scikit-learn==0.24.2
scipy==1.6.3
threadpoolctl==2.1.0
umap-learn==0.5.1
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the minimal Python reproduction using a 4097-row scipy sparse matrix and the traceback at pynndescent_.py line 1181. Compare sparse and dense fit_transform paths, then verify that fitting and pickling the sparse-input reducer succeeds without the reported numba error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100