lmcinnes / lmcinnes/pynndescent

Numba TypingError - too many positional arguments when trying to load saved umap models with v0.5.13

Open
#247 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
970
Forks
107
PR merge metrics
No merged PRs in 30d

Description

Version `0.5.13` with numba 0.60.0 and 0.59.1 present this error when trying to load saved umap pickles when running plain umap with Jaccard on scipy sparse vectors.

`pynndescent==0.5.13`
`numba==0.59.1 & 0.6.0` I receive this error on loading a pickle:
```

UMAP(angular_rp_forest=True, metric='jaccard', min_dist=0.1, n_jobs=1, n_neighbors=25, random_state=42, verbose=True)

...

with open('pickled_umap.pkl', 'wb') as fout:
pickle.dump(fit_model, fout)

with open('pickled_umap.pkl', 'rb') as fin:
foo = pickle.load(fin)

Building and compiling sparse search function
*** numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Internal error at .
too many positional arguments
During: resolving callee type: type(CPUDispatcher())
During: typing of call at ../lib/python3.10/site-packages/pynndescent/pynndescent_.py (1560)

Enable logging at debug level for details.

File "../../../../../../miniforge3/envs/chemspace_env/lib/python3.10/site-packages/pynndescent/pynndescent_.py", line 1560:
def search_closure(

d = np.float32(
dist(
^
```
(0.59.x is more detailed, 0.60.0 errors were not as helfpul)

With 0.5.12 and 0.5.11 and 0.59.1 the save and load work as expected.

A few other notes and observations.

I went down to pynndescent=0.5.1 and at that point umap is unimportable presenting pynndescent->numba related bootstrapping issues:

```Traceback (most recent call last):
File "~/umap-develop/src/umap/step/cluster/runner_umap.py", line 15, in
import umap
File "/home/ubuntu/miniforge3/envs/umap_env/lib/python3.10/site-packages/umap/__init__.py", line 2, in
from .umap_ import UMAP
File "/home/ubuntu/miniforge3/envs/umap_env/lib/python3.10/site-packages/umap/umap_.py", line 48, in
from pynndescent import NNDescent
File "/home/ubuntu/miniforge3/envs/umap_env/lib/python3.10/site-packages/pynndescent/__init__.py", line 3, in
from .pynndescent_ import NNDescent, PyNNDescentTransformer
File "/home/ubuntu/miniforge3/envs/umap_env/lib/python3.10/site-packages/pynndescent/pynndescent_.py", line 21, in
import pynndescent.sparse as sparse
File "/home/ubuntu/miniforge3/envs/umap_env/lib/python3.10/site-packages/pynndescent/sparse.py", line 10, in
from pynndescent.utils import norm, tau_rand
File "/home/ubuntu/miniforge3/envs/umap_env/lib/python3.10/site-packages/pynndescent/utils.py", line 630, in
def simple_heap_push(priorities, indices, p, n):
File "/home/ubuntu/miniforge3/envs/umap_env/lib/python3.10/site-packages/numba/core/decorators.py", line 236, in wrapper
disp.compile(sig)
File "/home/ubuntu/miniforge3/envs/umap_env/lib/python3.10/site-packages/numba/core/dispatcher.py", line 957, in compile
cres = self._compiler.compile(args, return_type)
File "/home/ubuntu/miniforge3/envs/umap_env/lib/python3.10/site-packages/numba/core/dispatcher.py", line 125, in compile
status, retval = self._compile_cached(args, return_type)
...

AssertionError: Failed in nopython mode pipeline (step: native lowering)
Storing i64 to ptr of i16 ('size'). FE type uint16
```

### Workaround
As of this writing, I am able to confirm and reliably pickle and joblib.dump()/load() with `pynndescent=0.5.12` and as `numba=0.60.0`, the current installation recipes from conda for `umap` install 0.5.13 and 0.60. Setting a minimum numba compatibility with pynndescent may help as well as seen from the other import issues and even pickle output issues I ran into during my testing. Numba is mentioned here as the nature of the error messages changed as I downgraded versions, but upon writing this issue, it seems like going from 59 to 60 did nothing to break things, except some behavior in messaging was changed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the pickle load with pynndescent 0.5.13, numba 0.59.1 or 0.60.0, and sparse Jaccard vectors. Inspect pynndescent_.py around line 1560, then compare the working 0.5.12 behavior with sparse.py and utils.py. Done means saved UMAP models load without the Numba TypingError while preserving compatibility with the reported versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.