MaartenGr / MaartenGr/BERTopic

How to merge topics automatically after getting the potential hierarchy of all topics

Open
#1,896 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

I have read this part of the official document:
https://maartengr.github.io/BERTopic/getting_started/hierarchicaltopics/hierarchicaltopics.html#visualizations:~:text=Merge%20topics,-%C2%B6
It is realy a great way to creat the potential hierarchical nature of topics and merge topics!
I have a further question, which is how to to merge topics automatically after getting the potential hierarchy of all topics?
For example, when I ran:
```python
from scipy.cluster import hierarchy as sch

linkage_function = lambda x: sch.linkage(x, 'single', optimal_ordering=True)
hierarchical_topics = model.hierarchical_topics(abstracts, linkage_function=linkage_function)
model.visualize_hierarchy(hierarchical_topics=hierarchical_topics)
```
The figure is bellow:
![image](https://github.com/MaartenGr/BERTopic/assets/102606613/40c4da53-ee71-469f-a9c6-36cb4d7f9410)
I think the horizontal axis is 'distance' between each topic.
The merge method in official document is specifying the indexes of the topics. How can I merge topics automatically if the 'distance' between two topics is less than a sertain number, such as 0.3?
My model is defined like this:
```python
umap_model = UMAP(n_neighbors=20, n_components=15, min_dist=0.0, metric='cosine', random_state=42)
cluster_model = KMeans(n_clusters=100, random_state=42)
vectorizer_model = CountVectorizer(stop_words="english")

seed_words = [
'materials','physical', ...
]
ctfidf_model = ClassTfidfTransformer(
seed_words=seed_words,
seed_multiplier=5
)
model = BERTopic(embedding_model='./multilingual-e5-large-instruct',
umap_model=umap_model,
min_topic_size=50,
ctfidf_model=ctfidf_model,
hdbscan_model=cluster_model,
vectorizer_model=vectorizer_model,
)
```
Vision:
```
Name: bertopic
Version: 0.16.0
Summary: BERTopic performs topic Modeling with state-of-the-art transformer models.
Home-page: https://github.com/MaartenGr/BERTopic
Author: Maarten P. Grootendorst
Author-email: maartengrootendorst@gmail.com
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the hierarchical topics documentation and the mentioned model.hierarchical_topics, model.visualize_hierarchy, and topic-merging entry points. Trace how the hierarchy exposes topic distances and define the expected threshold behavior; done means the requested automatic merge workflow is supported or clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.