MaartenGr / MaartenGr/BERTopic

Some topics have topic words starts with same letters or end with same letters

Open
#1,385 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Hi @MaartenGr, I want to thank you for developing bertopic, as it has been instrumental in the smooth progress of our project. And I am truly grateful for providing us such an active/progressive discussion and solutions.

I am dealing with 2 million documents and below is the main code:

`
from bertopic import BERTopic
from hdbscan import HDBSCAN
from umap import UMAP

import numpy as np

from sentence_transformers import SentenceTransformer
sentence_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = sentence_model.encode(docs, show_progress_bar=False)

from sklearn.feature_extraction.text import CountVectorizer

umap_model = UMAP(n_neighbors=15, n_components=5,
min_dist=0.0, metric='cosine', random_state=42, low_memory=True)

vectorizer_model = CountVectorizer(ngram_range=(1, 2), stop_words="english", min_df=100)
model = BERTopic(vectorizer_model=vectorizer_model, umap_model=umap_model, calculate_probabilities=False, nr_topics=500)

topics, probabilities = model.fit_transform(docs)
red_topics = model.reduce_outliers(docs, topics, strategy='c-tf-idf')
model.update_topics(docs, topics=red_topics, vectorizer_model=vectorizer_model)
`
And I found some of the topics are just the group of topic words starting with same letters as
(I made them up to just give some examples)
'xabvd, xser, xwesd, xrfde'
'jfrsd, jresa, jliok, joiun'
'dau' 'dan' 'daud'
or ends with same letters as
'calcium' 'valium' 'xxxxium'

Does this come from CountVectorizer? And is there any way to fix this?
Thank you so much!

Respectfully,
Ji Hyun

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided BERTopic example and inspect the CountVectorizer configuration, then trace how fit_transform and update_topics produce the displayed topic words. Compare the resulting vocabulary and topic representations with the reported prefixes and suffixes. Done means establishing whether the behavior originates in CountVectorizer or another named component and documenting a reproducible fix or configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.