MaartenGr / MaartenGr/BERTopic
Number of topics decreases significantly every run
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hello Maarten!
These are the configs I am using for my model to run on a dataset of news articles. When running the model with the default min_cluster_size, I get 200+ topics. When I run it the second time, I get five topics (for 7,500 documents). I tried it with different numbers for the min_cluster_size and for whatever number I give ( 30, 100) I get 3 topics.
hdbscan_model = HDBSCAN(min_cluster_size=20,prediction_data=True).
embedding_model = "sentence-transformers/all-MiniLM-L6-v2"
vectorizer_model = CountVectorizer(stop_words="english")
`topic_model = BERTopic(
top_n_words=15,
hdbscan_model=hdbscan_model,
umap_model=UMAP(),
vectorizer_model=vectorizer_model,
language='english',
verbose=True)`
Could you please tell me if I am doing something wrong? I am running the code in Google Colab and have used BERTopic before i this env but my results have never differed so much from one run to another.
Best regards,
Daiana
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the BERTopic run in Google Colab with the posted HDBSCAN, UMAP, CountVectorizer, embedding model, and dataset settings, then compare topic counts across repeated runs. Check whether the same inputs and random state are used each time. Done means identifying the cause of the change from 200+ topics to five or three and documenting a reliable reproduction or correction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100