MaartenGr / MaartenGr/BERTopic
Segmentation fault error and semaphore warning while training on MacPro Intel processor
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
### Have you searched existing issues? π
- [x] I have searched and found no existing issues
### Desribe the bug
Hello,
I am trying to train a simple BERTopic model using available GPU, but I keep running into a segmentation fault and a leaked semaphore warning, to the point that the code does not compile. I am fitting (.fit) a model on 5,000 representative short descriptions and calling .transform on 368,049 short descriptions. I believe the lack of RAM should not be a problem, since I have tested the code with a small subset of the data, and I still run into the same issue.
Thank you in advance!
### Reproduction
I am using Python 3.12.10, and training on a 2019 MacPro with the following specifications:
- Processor: 2,7 GHz 24-Core Intel Xeon W
- Memory: 192 GB 2933 MHz DDR4
- macOS 15.4
Here is the code I use:
```python
from bertopic import BERTopic
import numpy as np
import config as cfg
from data_loader import DataLoader
import utils
from topic_modeler import TopicModeler
from umap import UMAP
from visualizer import Visualizer
from sentence_transformers import SentenceTransformer
'''Make sure SUBSETTING is set to False in config'''
device = utils.get_device() # get device
print(f"Running on {device}")
data_loader = DataLoader(cfg)
topic_modeler = TopicModeler(cfg)
df, docs = data_loader.load_data()
subset = docs[0:5000]
print(df.shape)
print(len(subset))
topic_model = BERTopic(
embedding_model=topic_modeler.embedding_model,
vectorizer_model=topic_modeler.vectorizer_model,
language=topic_modeler.language,
calculate_probabilities=True,
verbose=True,
umap_model=UMAP(n_neighbors=15, n_components=5, min_dist=0.0, metric='cosine', low_memory=True),
min_topic_size=topic_modeler.min_topic_size,
hdbscan_model=topic_modeler.hdbscan_model if cfg.CUSTOM_HDBSCAN else None
)
topic_model.fit(subset)
embeddings = SentenceTransformer(cfg.EMBEDDING_MODEL).encode(docs)
np.save(cfg.MODELS_PATH / f'TEST_embeddings_{cfg.MODEL_NAME}.npy', embeddings)
topics, probs = topic_model.transform(docs, embeddings=embeddings)
```
This is the error I get:
```python
BERTopic_training/finetuning.py
Running on mps
No sentence-transformers model found with name sentence-transformers/all-MiniLM-L6-v2. Creating a new one with mean pooling.
Loading unlabelled dataset: all_missions β Shape: (368049, 11)
TRUNCATION DISABLED β Using original dataset format.
FINAL DATA SHAPE: (368049, 11)
(368049, 11)
5000
2025-05-20 12:44:35,637 - BERTopic - Embedding - Transforming documents to embeddings.
Batches: 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 157/157 [00:17<00:00, 9.21it/s]
2025-05-20 12:44:52,799 - BERTopic - Embedding - Completed β
2025-05-20 12:44:52,800 - BERTopic - Dimensionality - Fitting the dimensionality reduction algorithm
OMP: Info #271: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
zsh: segmentation fault (...)/lastenv/bin/python
/usr/local/Cellar/python@3.12/3.12.10/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d
```
Here are the dependencies used:
Package , Version
--------------------- -----------
adjustText 1.3.0
annotated-types 0.7.0
anyio 4.9.0
appdirs 1.4.4
bertopic 0.17.0
blis 1.2.1
bokeh 3.7.3
catalogue 2.0.10
certifi 2025.4.26
charset-normalizer 3.4.2
click 8.1.8
cloudpathlib 0.21.1
cloudpickle 3.1.1
colorcet 3.1.0
colorspacious 1.1.2
confection 0.1.5
contourpy 1.3.2
cycler 0.12.1
cymem 2.0.11
dask 2024.12.1
dask-expr 1.1.21
datamapplot 0.5.1
datashader 0.18.1
distributed 2024.12.1
docker-pycreds 0.4.0
fastapi 0.110.0
filelock 3.18.0
fonttools 4.58.0
fsspec 2025.3.2
geopandas 1.0.1
gitdb 4.0.12
GitPython 3.1.44
hdbscan 0.8.40
huggingface-hub 0.31.4
idna 3.10
imageio 2.37.0
importlib_resources 6.5.2
Jinja2 3.1.6
joblib 1.5.0
kiwisolver 1.4.8
langcodes 3.5.0
language_data 1.3.0
lazy_loader 0.4
llvmlite 0.44.0
locket 1.0.0
lz4 4.4.4
marisa-trie 1.2.1
markdown-it-py 3.0.0
MarkupSafe 3.0.2
matplotlib 3.10.3
mdurl 0.1.2
mpmath 1.3.0
msgpack 1.1.0
multipledispatch 1.0.0
murmurhash 1.0.12
narwhals 1.40.0
networkx 3.4.2
nltk 3.9.1
numba 0.61.2
numpy 1.26.4
packaging 25.0
pandas 2.2.3
param 2.2.0
partd 1.4.2
pillow 11.2.1
pip 25.0.1
platformdirs 4.3.8
plotly 6.1.0
preshed 3.0.9
protobuf 6.31.0
psutil 7.0.0
pyarrow 20.0.0
pyct 0.5.0
pydantic 2.11.4
pydantic_core 2.33.2
Pygments 2.19.1
pygris 0.2.0
pylabeladjust 0.1.13
pynndescent 0.5.13
pyogrio 0.11.0
pyparsing 3.2.3
pyproj 3.7.1
Pyqtree 1.0.0
python-dateutil 2.9.0.post0
pytz 2025.2
PyYAML 6.0.2
rcssmin 1.2.1
regex 2024.11.6
requests 2.32.3
rich 14.0.0
rjsmin 1.2.4
safetensors 0.5.3
scikit-image 0.25.2
scikit-learn 1.4.2
scipy 1.15.3
sentence-transformers 4.1.0
sentry-sdk 2.28.0
setproctitle 1.3.6
setuptools 80.7.1
shapely 2.1.1
shellingham 1.5.4
six 1.17.0
smart-open 7.1.0
smmap 5.0.2
sniffio 1.3.1
sortedcontainers 2.4.0
spacy 3.8.6
spacy-legacy 3.0.12
spacy-loggers 1.0.5
srsly 2.5.1
starlette 0.36.3
sympy 1.14.0
tblib 3.1.0
thinc 8.3.4
threadpoolctl 3.6.0
tifffile 2025.5.10
tokenizers 0.21.1
toolz 1.0.0
torch 2.2.2
tornado 6.5
tqdm 4.67.1
transformers 4.51.3
typer 0.15.4
typing_extensions 4.13.2
typing-inspection 0.4.0
tzdata 2025.2
umap-learn 0.5.7
urllib3 2.4.0
wandb 0.19.11
wasabi 1.1.3
weasel 0.4.1
wrapt 1.17.2
xarray 2025.4.0
xyzservices 2025.4.0
zict 3.0.0
### BERTopic Version
0.17.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up β it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the provided BERTopic.fit call with UMAP on the reported Python 3.12.10 and macOS 15.4 setup, then inspect the failure at the dimensionality-reduction stage. Compare the behavior with the supplied dependency versions and the subsequent SentenceTransformer.encode and topic_model.transform calls. Done means identifying a reproducible cause for the segmentation fault and leaked semaphore warning, with a verified fix or documented incompatibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, python, pytorch
- Domain
- machine-learning, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100