MaartenGr / MaartenGr/BERTopic

Number of topics decreases significantly every run

Open
#1,784 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Hello Maarten!

These are the configs I am using for my model to run on a dataset of news articles. When running the model with the default min_cluster_size, I get 200+ topics. When I run it the second time, I get five topics (for 7,500 documents). I tried it with different numbers for the min_cluster_size and for whatever number I give ( 30, 100) I get 3 topics.

hdbscan_model = HDBSCAN(min_cluster_size=20,prediction_data=True). embedding_model = "sentence-transformers/all-MiniLM-L6-v2" vectorizer_model = CountVectorizer(stop_words="english")

`topic_model = BERTopic(

top_n_words=15,
hdbscan_model=hdbscan_model,
umap_model=UMAP(),
vectorizer_model=vectorizer_model,
language='english',
verbose=True)`

Could you please tell me if I am doing something wrong? I am running the code in Google Colab and have used BERTopic before i this env but my results have never differed so much from one run to another.

Best regards,
Daiana

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the BERTopic run in Google Colab with the posted HDBSCAN, UMAP, CountVectorizer, embedding model, and dataset settings, then compare topic counts across repeated runs. Check whether the same inputs and random state are used each time. Done means identifying the cause of the change from 200+ topics to five or three and documenting a reliable reproduction or correction.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.