MaartenGr / MaartenGr/BERTopic
.merge_models() alters HDBSCAN clustering
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
### Have you searched existing issues? 🔎
- [x] I have searched and found no existing issues
### Desribe the bug
I have 2 bertopic model with HDBSCAN configured similarly trained on 2 different subsets of data. However, when these models are merged the resulting merged model defaults to BaseCluster and bypasses the clustering when calling .transform()
### Reproduction
```python
umap_mode1l = UMAP(n_components=25, metric='cosine', random_state=42)
vectorizer_model1 = CountVectorizer(stop_words="english")
model1 = BERTopic(umap_model=umap_model1,
vectorizer_model=vectorizer_model1,
calculate_probabilities=True,
verbose=True)
model1.fit(data1, embeddings=embeddings1)
umap_model2 = UMAP(n_components=25, metric='cosine', random_state=42)
vectorizer_model2 = CountVectorizer(stop_words="english")
model2 = BERTopic(umap_model=umap_model2,
vectorizer_model=vectorizer_model2,
calculate_probabilities=True,
verbose=True)
model2.fit(data2, embeddings=embeddings2)
merged_model = BERTopic.merge_models([model1, model2],
min_similarity=0.7)
merged_model.hdbscan_model
```
### BERTopic Version
0.17.3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the report with the provided BERTopic.merge_models([model1, model2], min_similarity=0.7) example in BERTopic 0.17.3. Inspect merged_model.hdbscan_model and then follow the .transform() call to determine why merging replaces or bypasses HDBSCAN. Done means a merged model preserves the expected clustering behavior during transform.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100