MaartenGr / MaartenGr/BERTopic
Importing cuML`s UMAP before bertopic leads to FileNotFoundErrors
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
# Overview:
Running `from bertopic import BERTopic` after importing cuML's UMAP gives a `FileNotFoundError`. Essentially, it is looking for `numba==0.56.1`, which is not installed on my system. Re-running the same code block fails multiple times, each time looking for a different not-installed package, until it finally succeeds, with no distinguishable poor results after success.
This could also be an issue on UMAP's end, not on BERTopics, but there is some overlap. Especially when I want to use cuML's UMAP and not the native UMAP (which is when this happens).
# MVE
The MVE is found [here](https://colab.research.google.com/drive/1H3I2cwpnnK8UHdy6aVj76FAxNKQ_RR8V?usp=sharing).
# Why this happens:
I'm not 100% sure on this one. What I think is going on is that, while importing the BERTopic class, the `bertopic/_bertopic.py` file calls `from umap import UMAP`, which is the line that's failing. This issue does *not* occur when just running `from bertopic import BERTopic` (without importing cuML`s UMAP). Could it be that there's some dependency conflicts with those two?
Note: this also does not happen for cuML's HDBSCAN, just UMAP. Could that be because BERTopic is importing `import hdbscan` instead of `from hdbscan import HDBSCAN`?
The error occurs multiple times until it finally goes away (and who said just re-running the code doesn't work!). Here are the errors in order:
```python
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.10/dist-packages/numba-0.56.4.dist-info/METADATA'
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.10/dist-packages/scikit_learn-1.2.2.dist-info/METADATA'
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.10/dist-packages/llvmlite-0.39.1.dist-info/METADATA'
```
# Possible solution:
Figure out whether the user has imported native umap or cuML's umap, then import that one in the `bertopic/_bertopic.py` file. Installing those specific versions that umap is looking for runs into dependency errors.
# Workarounds:
Continually re-run `from bertopic import BERTopic` until it works. This is obviously a bad solution, but I have the following code locally:
```python
while True:
try:
from bertopic import BERTopic
break
except FileNotFoundError:
continue
except Exception as e:
print(e)
break
```
# Full stack trace (for the first time the error occurs):
```python
/usr/local/lib/python3.10/dist-packages/umap/distances.py:1063: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
/usr/local/lib/python3.10/dist-packages/umap/distances.py:1071: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
/usr/local/lib/python3.10/dist-packages/umap/distances.py:1086: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
/usr/local/lib/python3.10/dist-packages/umap/umap_.py:660: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
@numba.jit()
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py](https://localhost:8080/#) in _dep_map(self)
3107 try:
-> 3108 return self.__dep_map
3109 except AttributeError:
16 frames
[/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py](https://localhost:8080/#) in __getattr__(self, attr)
2900 if attr.startswith('_'):
-> 2901 raise AttributeError(attr)
2902 return getattr(self._provider, attr)
AttributeError: _DistInfoDistribution__dep_map
During handling of the above exception, another exception occurred:
AttributeError Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py](https://localhost:8080/#) in _parsed_pkg_info(self)
3098 try:
-> 3099 return self._pkg_info
3100 except AttributeError:
[/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py](https://localhost:8080/#) in __getattr__(self, attr)
2900 if attr.startswith('_'):
-> 2901 raise AttributeError(attr)
2902 return getattr(self._provider, attr)
AttributeError: _pkg_info
During handling of the above exception, another exception occurred:
FileNotFoundError Traceback (most recent call last)
[](https://localhost:8080/#) in ()
4 # from cuml.cluster import HDBSCAN
5
----> 6 from bertopic import BERTopic
[/usr/local/lib/python3.10/dist-packages/bertopic/__init__.py](https://localhost:8080/#) in
----> 1 from bertopic._bertopic import BERTopic
2
3 __version__ = "0.15.0"
4
5 __all__ = [
[/usr/local/lib/python3.10/dist-packages/bertopic/_bertopic.py](https://localhost:8080/#) in
36 # Models
37 import hdbscan
---> 38 from umap import UMAP
39 from sklearn.preprocessing import normalize
40 from sklearn import __version__ as sklearn_version
[/usr/local/lib/python3.10/dist-packages/umap/__init__.py](https://localhost:8080/#) in
37
38 try:
---> 39 __version__ = pkg_resources.get_distribution("umap-learn").version
40 except pkg_resources.DistributionNotFound:
41 __version__ = "0.5-dev"
[/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py](https://localhost:8080/#) in get_distribution(dist)
524 dist = Requirement.parse(dist)
525 if isinstance(dist, Requirement):
--> 526 dist = get_provider(dist)
527 if not isinstance(dist, Distribution):
528 raise TypeError("Expected string, Requirement, or Distribution", dist)
[/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py](https://localhost:8080/#) in get_provider(moduleOrReq)
396 """Return an IResourceProvider for the named module or requirement"""
397 if isinstance(moduleOrReq, Requirement):
--> 398 return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
399 try:
400 module = sys.modules[moduleOrReq]
[/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py](https://localhost:8080/#) in require(self, *requirements)
964 included, even if they were already activated in this working set.
965 """
--> 966 needed = self.resolve(parse_requirements(requirements))
967
968 for dist in needed:
[/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py](https://localhost:8080/#) in resolve(self, requirements, env, installer, replace_conflicting, extras)
830
831 # push the new requirements onto the stack
--> 832 new_requirements = dist.requires(req.extras)[::-1]
833 requirements.extend(new_requirements)
834
[/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py](https://localhost:8080/#) in requires(self, extras)
2819 def requires(self, extras=()):
2820 """List of Requirements needed for this distro if `extras` are used"""
-> 2821 dm = self._dep_map
2822 deps = []
2823 deps.extend(dm.get(None, ()))
[/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py](https://localhost:8080/#) in _dep_map(self)
3108 return self.__dep_map
3109 except AttributeError:
-> 3110 self.__dep_map = self._compute_dependencies()
3111 return self.__dep_map
3112
[/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py](https://localhost:8080/#) in _compute_dependencies(self)
3117 reqs = []
3118 # Including any condition expressions
-> 3119 for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
3120 reqs.extend(parse_requirements(req))
3121
[/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py](https://localhost:8080/#) in _parsed_pkg_info(self)
3099 return self._pkg_info
3100 except AttributeError:
-> 3101 metadata = self.get_metadata(self.PKG_INFO)
3102 self._pkg_info = email.parser.Parser().parsestr(metadata)
3103 return self._pkg_info
[/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py](https://localhost:8080/#) in get_metadata(self, name)
1515 return ""
1516 path = self._get_metadata_path(name)
-> 1517 value = self._get(path)
1518 try:
1519 return value.decode('utf-8')
[/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py](https://localhost:8080/#) in _get(self, path)
1724
1725 def _get(self, path):
-> 1726 with open(path, 'rb') as stream:
1727 return stream.read()
1728
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.10/dist-packages/numba-0.56.4.dist-info/METADATA'
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the linked Colab MVE with cuML's UMAP imported before BERTopic. Inspect bertopic/_bertopic.py at the `from umap import UMAP` import and compare it with the reported dependency behavior; done means BERTopic imports without the repeated FileNotFoundError in that setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100