MaartenGr / MaartenGr/BERTopic
loading BERTopic model error 'NNDescent' object has no attribute '_bit_trees'
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
I saved a model about two weeks ago but as of today can't load the model. This is the error message I get. Thank you for your help!
AttributeError Traceback (most recent call last)
Cell In[11], line 2
1 from bertopic import BERTopic
----> 2 topic_model81_100 = BERTopic.load("bertopic_model_81-100.model")
File [/opt/mamba/lib/python3.11/site-packages/bertopic/_bertopic.py:3047](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/site-packages/bertopic/_bertopic.py#line=3046), in BERTopic.load(cls, path, embedding_model)
3045 topic_model.embedding_model = select_backend(embedding_model)
3046 else:
-> 3047 topic_model = joblib.load(file)
3048 return topic_model
3050 # Load from directory or HF
File [/opt/mamba/lib/python3.11/site-packages/joblib/numpy_pickle.py:648](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/site-packages/joblib/numpy_pickle.py#line=647), in load(filename, mmap_mode)
646 filename = getattr(fobj, 'name', '')
647 with _read_fileobject(fobj, filename, mmap_mode) as fobj:
--> 648 obj = _unpickle(fobj)
649 else:
650 with open(filename, 'rb') as f:
File [/opt/mamba/lib/python3.11/site-packages/joblib/numpy_pickle.py:577](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/site-packages/joblib/numpy_pickle.py#line=576), in _unpickle(fobj, filename, mmap_mode)
575 obj = None
576 try:
--> 577 obj = unpickler.load()
578 if unpickler.compat_mode:
579 warnings.warn("The file '%s' has been generated with a "
580 "joblib version less than 0.10. "
581 "Please regenerate this pickle file."
582 % filename,
583 DeprecationWarning, stacklevel=3)
File [/opt/mamba/lib/python3.11/pickle.py:1213](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/pickle.py#line=1212), in _Unpickler.load(self)
1211 raise EOFError
1212 assert isinstance(key, bytes_types)
-> 1213 dispatch[key[0]](self)
1214 except _Stop as stopinst:
1215 return stopinst.value
File [/opt/mamba/lib/python3.11/site-packages/joblib/numpy_pickle.py:402](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/site-packages/joblib/numpy_pickle.py#line=401), in NumpyUnpickler.load_build(self)
394 def load_build(self):
395 """Called to set the state of a newly created object.
396
397 We capture it to replace our place-holder objects, NDArrayWrapper or
(...)
400 NDArrayWrapper is used for backward compatibility with joblib <= 0.9.
401 """
--> 402 Unpickler.load_build(self)
404 # For backward compatibility, we support NDArrayWrapper objects.
405 if isinstance(self.stack[-1], (NDArrayWrapper, NumpyArrayWrapper)):
File [/opt/mamba/lib/python3.11/pickle.py:1718](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/pickle.py#line=1717), in _Unpickler.load_build(self)
1716 setstate = getattr(inst, "__setstate__", None)
1717 if setstate is not None:
-> 1718 setstate(state)
1719 return
1720 slotstate = None
File [/opt/mamba/lib/python3.11/site-packages/pynndescent/pynndescent_.py:994](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/site-packages/pynndescent/pynndescent_.py#line=993), in NNDescent.__setstate__(self, d)
992 self._init_sparse_search_function()
993 else:
--> 994 self._init_search_function()
File [/opt/mamba/lib/python3.11/site-packages/pynndescent/pynndescent_.py:1228](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/site-packages/pynndescent/pynndescent_.py#line=1227), in NNDescent._init_search_function(self)
1225 tree_indices = self._search_forest[0].indices
1226 tree_children = self._search_forest[0].children
-> 1228 if self._bit_trees:
1229 @numba.njit(
1230 [
1231 numba.types.Array(numba.types.int32, 1, "C", readonly=True)(
(...)
1237 )
1238 def tree_search_closure(point, rng_state):
1239 node = 0
AttributeError: 'NNDescent' object has no attribute '_bit_trees'
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with BERTopic.load in bertopic/_bertopic.py and the NNDescent.__setstate__ and _init_search_function paths shown in pynndescent_.py. Compare the saved model's and current dependency environments, then verify that loading the saved model completes without the _bit_trees AttributeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100