MaartenGr / MaartenGr/BERTopic
Merge Multiple Fitted Models - Error on Viewing Hierarchical Topics
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
**Problem Overview**
When using the ```merged_model = BERTopic.merge_models([topic_model_1, topic_model_2])``` command the produced merged topic model cannot be visualised as a hierarchical topic model anymore, even if the constituent models can be.
**Error Code**
```
hierarchical_topics = merged_model.hierarchical_topics(docs, linkage_function = linkage_function)
Traceback (most recent call last):
Cell In[14], line 1
hierarchical_topics = merged_model.hierarchical_topics(docs, linkage_function = linkage_function)
File ~/anaconda3/envs/tf/lib/python3.9/site-packages/bertopic/_bertopic.py:975 in hierarchical_topics
embeddings = self.c_tf_idf_[self._outliers:]
TypeError: 'NoneType' object is not subscriptable
```
**Minimum Working Example**
```python
from umap import UMAP
from bertopic import BERTopic
from datasets import load_dataset
from sklearn.datasets import fetch_20newsgroups
docs = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))["data"]
# Create topic models
umap_model = UMAP(n_neighbors=15, n_components=5, min_dist=0.0, metric='cosine', random_state=42)
topic_model_1 = BERTopic(umap_model=umap_model, min_topic_size=20).fit(docs[0:1000])
topic_model_2 = BERTopic(umap_model=umap_model, min_topic_size=20).fit(docs[1000:2000])
# Combine all models into one
merged_model = BERTopic.merge_models([topic_model_1, topic_model_2])
# #Visualise Hierarchical Topic Model
linkage_function = lambda x: sch.linkage(x, 'ward', optimal_ordering=True)
#Use fitted model to extract hierarchies
hierarchical_topics = merged_model.hierarchical_topics(docs, linkage_function = linkage_function)
#Visualise Hierarchies
fig = merged_model.visualize_hierarchy(hierarchical_topics=hierarchical_topics)
fig.write_html("merged_model.html")
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the provided minimum working example and inspect _bertopic.py at hierarchical_topics, especially the c-TF-IDF access shown in the traceback, alongside merge_models. Done means a model produced by merge_models can run hierarchical_topics and then visualize_hierarchy without the reported NoneType error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100