MaartenGr / MaartenGr/BERTopic
Kernel crash from .fit_transform
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
I am trying to run the BERTopic model on a Mac (16 GB RAM, Intel i7) using example data from sklearn.datasets (fetch_20newsgroups). The following lines of code are run without any problems.
```
topic_model = BERTopic()
docs = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))['data']
```
But when running this line the kernel crashes:
```
topics, probs = topic_model.fit_transform(docs)
```
The problem seems to occur towards the end, because it happens after 100 % of the documents are transformed into embeddings and it says embedding - completed. There is the following warning message before the kernel dies:
```
OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Restarting kernel...
```
The same happens when running it on just a few and short documents (even as short as sentences).
Do you know what the problem might be and how to fix it?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the crash with BERTopic(), fetch_20newsgroups, and topic_model.fit_transform(docs), including the short-document case. Examine the OMP and resource_tracker warnings and the point after embedding completes; done means fit_transform finishes without restarting the kernel.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100