MaartenGr / MaartenGr/BERTopic

Merge Multiple Fitted Models - Error on Viewing Hierarchical Topics

Open
#1,682 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

**Problem Overview**
When using the ```merged_model = BERTopic.merge_models([topic_model_1, topic_model_2])``` command the produced merged topic model cannot be visualised as a hierarchical topic model anymore, even if the constituent models can be.

**Error Code**
```
hierarchical_topics = merged_model.hierarchical_topics(docs, linkage_function = linkage_function)
Traceback (most recent call last):

Cell In[14], line 1
hierarchical_topics = merged_model.hierarchical_topics(docs, linkage_function = linkage_function)

File ~/anaconda3/envs/tf/lib/python3.9/site-packages/bertopic/_bertopic.py:975 in hierarchical_topics
embeddings = self.c_tf_idf_[self._outliers:]

TypeError: 'NoneType' object is not subscriptable
```

**Minimum Working Example**

```python
from umap import UMAP
from bertopic import BERTopic
from datasets import load_dataset
from sklearn.datasets import fetch_20newsgroups

docs = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes'))["data"]

# Create topic models
umap_model = UMAP(n_neighbors=15, n_components=5, min_dist=0.0, metric='cosine', random_state=42)
topic_model_1 = BERTopic(umap_model=umap_model, min_topic_size=20).fit(docs[0:1000])
topic_model_2 = BERTopic(umap_model=umap_model, min_topic_size=20).fit(docs[1000:2000])

# Combine all models into one
merged_model = BERTopic.merge_models([topic_model_1, topic_model_2])

# #Visualise Hierarchical Topic Model
linkage_function = lambda x: sch.linkage(x, 'ward', optimal_ordering=True)

#Use fitted model to extract hierarchies
hierarchical_topics = merged_model.hierarchical_topics(docs, linkage_function = linkage_function)

#Visualise Hierarchies
fig = merged_model.visualize_hierarchy(hierarchical_topics=hierarchical_topics)
fig.write_html("merged_model.html")
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided minimum working example and inspect _bertopic.py at hierarchical_topics, especially the c-TF-IDF access shown in the traceback, alongside merge_models. Done means a model produced by merge_models can run hierarchical_topics and then visualize_hierarchy without the reported NoneType error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.