MaartenGr / MaartenGr/BERTopic

Kernel crash from .fit_transform

Open
#1,765 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

I am trying to run the BERTopic model on a Mac (16 GB RAM, Intel i7) using example data from sklearn.datasets (fetch_20newsgroups). The following lines of code are run without any problems.

```
topic_model = BERTopic()
docs = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))['data']
```

But when running this line the kernel crashes:
```
topics, probs = topic_model.fit_transform(docs)
```
The problem seems to occur towards the end, because it happens after 100 % of the documents are transformed into embeddings and it says embedding - completed. There is the following warning message before the kernel dies:
```
OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

Restarting kernel...
```

The same happens when running it on just a few and short documents (even as short as sentences).

Do you know what the problem might be and how to fix it?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the crash with BERTopic(), fetch_20newsgroups, and topic_model.fit_transform(docs), including the short-document case. Examine the OMP and resource_tracker warnings and the point after embedding completes; done means fit_transform finishes without restarting the kernel.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.