Using BERTopic on Chinese and Japanese Texts

Hello Maarten, there is one thing I would like to mention when using BERTopic to analyze Chinese and Japanese texts. If we run the following code to analyze Chinese or Japanese:

from bertopic import BERTopic
topic_model_multi = BERTopic(language="multilingual", calculate_probabilities=True, verbose=True)
topics_multi, probs_multi = topic_model_multi.fit_transform(texts)

We will get the results that look like the following (in case of analyzing Japanese texts):

9_おめでとう_おはようございます_おはありです_ありがとうございます
10_10552100本祭は雨天時は29日へ順延タイムスケジュール等は後日流れるのでお待ちく...
11_野党がひたすら揚げ足取りをしているというのはどういうことでしょうか私は今通常国会開会か...

However, the topics here are more like sentences than words.

After some research, some online posts indicate that for texts that are not separated by spaces like English or French are, we need to first transform the texts into a similar English text format (with space between words), and then use BERTopic to analyze the transformed text. Hence, we cannot initiate BERTopic on Chinese or Japanese (texts that are not separated by spaces) right out of the box without any preprocessing.

I would like to ask your opinion on this issue. If you agree on what were being discussed, could you make a reminder so that more people will know about this?

Thanks a lot!

Start by reviewing BERTopic's multilingual-language behavior and the discussion in this issue, focusing on how Chinese and Japanese text is tokenized before topic representation. Done means documenting whether preprocessing is required and clearly explaining the supported usage or limitation for these languages.