[Feature]: Add custom token corpus for other languages for bm25

What feature would you like to request?

Hi, I’m exploring how to enable Qdrant/bm25 to properly tokenize and vectorize the Thai language.

I noticed that the model cache directory contains multiple language-specific text files:

(see image below)

I’d like to add a custom Thai corpus based on this word list:
https://github.com/PyThaiNLP/pythainlp/blob/dev/pythainlp/corpus/words_th.txt

However, simply adding a thai.txt file to the model cache directory does not work dynamically (on the fly). Could you clarify the correct way to extend or register a new language corpus for BM25, or whether additional configuration or rebuilding steps are required?

Is there any additional information you would like to provide?

No response

Start by inspecting BM25's language-corpus registration and model-cache loading, then compare the existing language-specific text files with PyThaiNLP's words_th.txt corpus. Determine whether adding Thai requires configuration, a rebuild, or a code change, and define how successful Thai tokenization and vectorization should be verified.