MaartenGr / MaartenGr/BERTopic
Some topics have topic words starts with same letters or end with same letters
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hi @MaartenGr, I want to thank you for developing bertopic, as it has been instrumental in the smooth progress of our project. And I am truly grateful for providing us such an active/progressive discussion and solutions.
I am dealing with 2 million documents and below is the main code:
`
from bertopic import BERTopic
from hdbscan import HDBSCAN
from umap import UMAP
import numpy as np
from sentence_transformers import SentenceTransformer
sentence_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = sentence_model.encode(docs, show_progress_bar=False)
from sklearn.feature_extraction.text import CountVectorizer
umap_model = UMAP(n_neighbors=15, n_components=5,
min_dist=0.0, metric='cosine', random_state=42, low_memory=True)
vectorizer_model = CountVectorizer(ngram_range=(1, 2), stop_words="english", min_df=100)
model = BERTopic(vectorizer_model=vectorizer_model, umap_model=umap_model, calculate_probabilities=False, nr_topics=500)
topics, probabilities = model.fit_transform(docs)
red_topics = model.reduce_outliers(docs, topics, strategy='c-tf-idf')
model.update_topics(docs, topics=red_topics, vectorizer_model=vectorizer_model)
`
And I found some of the topics are just the group of topic words starting with same letters as
(I made them up to just give some examples)
'xabvd, xser, xwesd, xrfde'
'jfrsd, jresa, jliok, joiun'
'dau' 'dan' 'daud'
or ends with same letters as
'calcium' 'valium' 'xxxxium'
Does this come from CountVectorizer? And is there any way to fix this?
Thank you so much!
Respectfully,
Ji Hyun
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the provided BERTopic example and inspect the CountVectorizer configuration, then trace how fit_transform and update_topics produce the displayed topic words. Compare the resulting vocabulary and topic representations with the reported prefixes and suffixes. Done means establishing whether the behavior originates in CountVectorizer or another named component and documenting a reproducible fix or configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100