MaartenGr / MaartenGr/BERTopic

Which hyper parameter mostly influence the number of topics for Chinese texts?

Open
#1,998 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

```
import jieba
def tokenize_zh(text):
words = jieba.lcut(text)
words = list(filter(lambda x: (len(x)>1), words))
return words

import numpy as np
from umap import UMAP
from sklearn.feature_extraction.text import CountVectorizer
from sentence_transformers import SentenceTransformer
embedding_model = SentenceTransformer("BAAI/bge-base-zh-v1.5") # a Chinese embedding model
vectorizer_model = CountVectorizer(tokenizer=tokenize_zh, stop_words=stopwords, ngram_range=(1, 3), min_df=3) # The variiable stopwords is a list.
umap_model = UMAP(n_neighbors=15, n_components=5, min_dist=0.0, metric='cosine', random_state=42)
topic_model = BERTopic(language='chinese', embedding_model=embedding_model, umap_model=umap_model, top_n_words=10, min_topic_size=10, n_gram_range=(1,3), vectorizer_model=vectorizer_model, calculate_probabilities=True, verbose=True)

topics, probs = topic_model.fit_transform(docs)
topic_model.get_topic_info()
```

It has only 3 topics as below, much less than the Mallet tool.
![image](https://github.com/MaartenGr/BERTopic/assets/1741341/9d84ac37-e59f-4186-980c-64d6ee571661)

I almost test every hyper parameters here, at last found the `n_neighbors `in the `UMAP `function works most apparently, however, even changing it from 15 to 50 only adds 1 new topic as below:
![image](https://github.com/MaartenGr/BERTopic/assets/1741341/3a853821-1981-429c-b274-e2b622a1810a)

It seems BERTopic need more tuning parameters for Chinese or even CJK texts. Can anyone share some experience, please?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied Chinese example and inspect how BERTopic applies the UMAP, CountVectorizer, and topic-size settings during fit_transform. Reproduce the topic-count changes while varying the named parameters; done would require a documented, reproducible explanation or a clearly scoped improvement for Chinese or CJK behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.