MaartenGr / MaartenGr/BERTopic

loading BERTopic model error 'NNDescent' object has no attribute '_bit_trees'

Open
#1,905 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

I saved a model about two weeks ago but as of today can't load the model. This is the error message I get. Thank you for your help!
AttributeError Traceback (most recent call last)
Cell In[11], line 2
1 from bertopic import BERTopic
----> 2 topic_model81_100 = BERTopic.load("bertopic_model_81-100.model")

File [/opt/mamba/lib/python3.11/site-packages/bertopic/_bertopic.py:3047](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/site-packages/bertopic/_bertopic.py#line=3046), in BERTopic.load(cls, path, embedding_model)
3045 topic_model.embedding_model = select_backend(embedding_model)
3046 else:
-> 3047 topic_model = joblib.load(file)
3048 return topic_model
3050 # Load from directory or HF

File [/opt/mamba/lib/python3.11/site-packages/joblib/numpy_pickle.py:648](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/site-packages/joblib/numpy_pickle.py#line=647), in load(filename, mmap_mode)
646 filename = getattr(fobj, 'name', '')
647 with _read_fileobject(fobj, filename, mmap_mode) as fobj:
--> 648 obj = _unpickle(fobj)
649 else:
650 with open(filename, 'rb') as f:

File [/opt/mamba/lib/python3.11/site-packages/joblib/numpy_pickle.py:577](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/site-packages/joblib/numpy_pickle.py#line=576), in _unpickle(fobj, filename, mmap_mode)
575 obj = None
576 try:
--> 577 obj = unpickler.load()
578 if unpickler.compat_mode:
579 warnings.warn("The file '%s' has been generated with a "
580 "joblib version less than 0.10. "
581 "Please regenerate this pickle file."
582 % filename,
583 DeprecationWarning, stacklevel=3)

File [/opt/mamba/lib/python3.11/pickle.py:1213](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/pickle.py#line=1212), in _Unpickler.load(self)
1211 raise EOFError
1212 assert isinstance(key, bytes_types)
-> 1213 dispatch[key[0]](self)
1214 except _Stop as stopinst:
1215 return stopinst.value

File [/opt/mamba/lib/python3.11/site-packages/joblib/numpy_pickle.py:402](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/site-packages/joblib/numpy_pickle.py#line=401), in NumpyUnpickler.load_build(self)
394 def load_build(self):
395 """Called to set the state of a newly created object.
396
397 We capture it to replace our place-holder objects, NDArrayWrapper or
(...)
400 NDArrayWrapper is used for backward compatibility with joblib <= 0.9.
401 """
--> 402 Unpickler.load_build(self)
404 # For backward compatibility, we support NDArrayWrapper objects.
405 if isinstance(self.stack[-1], (NDArrayWrapper, NumpyArrayWrapper)):

File [/opt/mamba/lib/python3.11/pickle.py:1718](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/pickle.py#line=1717), in _Unpickler.load_build(self)
1716 setstate = getattr(inst, "__setstate__", None)
1717 if setstate is not None:
-> 1718 setstate(state)
1719 return
1720 slotstate = None

File [/opt/mamba/lib/python3.11/site-packages/pynndescent/pynndescent_.py:994](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/site-packages/pynndescent/pynndescent_.py#line=993), in NNDescent.__setstate__(self, d)
992 self._init_sparse_search_function()
993 else:
--> 994 self._init_search_function()

File [/opt/mamba/lib/python3.11/site-packages/pynndescent/pynndescent_.py:1228](https://user-fabiandrft-9118-0.user.lab.sspcloud.fr/opt/mamba/lib/python3.11/site-packages/pynndescent/pynndescent_.py#line=1227), in NNDescent._init_search_function(self)
1225 tree_indices = self._search_forest[0].indices
1226 tree_children = self._search_forest[0].children
-> 1228 if self._bit_trees:
1229 @numba.njit(
1230 [
1231 numba.types.Array(numba.types.int32, 1, "C", readonly=True)(
(...)
1237 )
1238 def tree_search_closure(point, rng_state):
1239 node = 0

AttributeError: 'NNDescent' object has no attribute '_bit_trees'

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with BERTopic.load in bertopic/_bertopic.py and the NNDescent.__setstate__ and _init_search_function paths shown in pynndescent_.py. Compare the saved model's and current dependency environments, then verify that loading the saved model completes without the _bit_trees AttributeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.