function "preprocess" in file "convert_corpus.py"

after "text=preprocess(text)"; some Chinese character change to garbled. such as "充满" to "？满"， “？” for garbled. Is there something wrong? I think this function is to normalize all num to the number "0" and all English word to "X" and delet space "".

Start by reading the preprocess function in convert_corpus.py and reproduce the reported conversion of “充满” to “？满” after text=preprocess(text). Check which normalization step affects Chinese characters; done means Chinese text remains intact while the stated number, English-word, and space normalization still behaves as intended.

Written by the indexing model from the issue text.