MaartenGr / MaartenGr/BERTopic
Which hyper parameter mostly influence the number of topics for Chinese texts?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
```
import jieba
def tokenize_zh(text):
words = jieba.lcut(text)
words = list(filter(lambda x: (len(x)>1), words))
return words
import numpy as np
from umap import UMAP
from sklearn.feature_extraction.text import CountVectorizer
from sentence_transformers import SentenceTransformer
embedding_model = SentenceTransformer("BAAI/bge-base-zh-v1.5") # a Chinese embedding model
vectorizer_model = CountVectorizer(tokenizer=tokenize_zh, stop_words=stopwords, ngram_range=(1, 3), min_df=3) # The variiable stopwords is a list.
umap_model = UMAP(n_neighbors=15, n_components=5, min_dist=0.0, metric='cosine', random_state=42)
topic_model = BERTopic(language='chinese', embedding_model=embedding_model, umap_model=umap_model, top_n_words=10, min_topic_size=10, n_gram_range=(1,3), vectorizer_model=vectorizer_model, calculate_probabilities=True, verbose=True)
topics, probs = topic_model.fit_transform(docs)
topic_model.get_topic_info()
```
It has only 3 topics as below, much less than the Mallet tool.

I almost test every hyper parameters here, at last found the `n_neighbors `in the `UMAP `function works most apparently, however, even changing it from 15 to 50 only adds 1 new topic as below:

It seems BERTopic need more tuning parameters for Chinese or even CJK texts. Can anyone share some experience, please?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the supplied Chinese example and inspect how BERTopic applies the UMAP, CountVectorizer, and topic-size settings during fit_transform. Reproduce the topic-count changes while varying the named parameters; done would require a documented, reproducible explanation or a clearly scoped improvement for Chinese or CJK behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100