lmcinnes / lmcinnes/umap

UMAP model load complication with joblib and numba

Open
#473 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.3k
Forks
871
Avg merge
1d 13h
Merged PRs (30d)
5

Description

I face a problem in loading a dumped model by joblib which is not similar to the previous raised issues, as far as I understood.
Here are the version of different relevant modules:

pathon version 3.7.7
umap version 0.4.4
pynndescent version 0.4.7
numba version 0.46.0
llvmlite version 0.30.0

And the snippet of error message:

Traceback (most recent call last):

File "", line 1, in
umap_model = joblib.load( model_dir + umap_reducer_fname )

File "C:\Users\ansa_ho\Anaconda3\lib\site-packages\joblib\numpy_pickle.py", line 585, in load
obj = _unpickle(fobj, filename, mmap_mode)

File "C:\Users\ansa_ho\Anaconda3\lib\site-packages\joblib\numpy_pickle.py", line 504, in _unpickle
obj = unpickler.load()

File "C:\Users\ansa_ho\Anaconda3\lib\pickle.py", line 1088, in load
dispatchkey[0]

File "C:\Users\ansa_ho\Anaconda3\lib\site-packages\joblib\numpy_pickle.py", line 329, in load_build
Unpickler.load_build(self)

File "C:\Users\ansa_ho\Anaconda3\lib\pickle.py", line 1552, in load_build
setstate(state)

File "C:\Users\ansa_ho\Anaconda3\lib\site-packages\pynndescent\pynndescent_.py", line 1026, in setstate
self._rp_forest = tuple([renumbaify_tree(tree) for tree in d["_rp_forest"]])

File "C:\Users\ansa_ho\Anaconda3\lib\site-packages\pynndescent\pynndescent_.py", line 1026, in
self._rp_forest = tuple([renumbaify_tree(tree) for tree in d["_rp_forest"]])

File "C:\Users\ansa_ho\Anaconda3\lib\site-packages\pynndescent\rp_trees.py", line 1178, in renumbaify_tree
hyperplanes.extend(tree.hyperplanes)

File "C:\Users\ansa_ho\Anaconda3\lib\site-packages\numba\typed\typedlist.py", line 265, in extend
return _extend(self, iterable)

File "C:\Users\ansa_ho\Anaconda3\lib\site-packages\numba\dispatcher.py", line 401, in _compile_for_args
error_rewrite(e, 'typing')

File "C:\Users\ansa_ho\Anaconda3\lib\site-packages\numba\dispatcher.py", line 344, in error_rewrite
reraise(type(e), e, None)

File "C:\Users\ansa_ho\Anaconda3\lib\site-packages\numba\six.py", line 668, in reraise
raise value.with_traceback(tb)

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Internal error at <numba.typeinfer.CallConstraint object at 0x0000023BCE0EA2C8>.
Failed in nopython mode pipeline (step: nopython mode backend)

File "......\Anaconda3\lib\site-packages\numba\listobject.py", line 434:
def impl(l, item):
casteditem = _cast(item, itemty)
^

[1] During: lowering "$0.4 = call $0.1(item, $0.3, func=$0.1, args=[Var(item, C:\Users\ansa_ho\Anaconda3\lib\site-packages\numba\listobject.py (434)), Var($0.3, C:\Users\ansa_ho\Anaconda3\lib\site-packages\numba\listobject.py (434))], kws=(), vararg=None)" at C:\Users\ansa_ho\Anaconda3\lib\site-packages\numba\listobject.py (434)
[2] During: resolving callee type: BoundFunction((<class 'numba.types.containers.ListType'>, 'append') for ListType[array(float64, 2d, C)])
[3] During: typing of call at C:\Users\ansa_ho\Anaconda3\lib\site-packages\numba\listobject.py (840)


I'd appreciate any support.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the joblib.load failure with the listed Python, UMAP, pynndescent, numba, and llvmlite versions. Then inspect pynndescent/pynndescent_.py at setstate and pynndescent/rp_trees.py at renumbaify_tree, where the traceback fails. Done means identifying a confirmed compatibility fix or a clear reproduction and resolution path for loading the dumped model.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.