PyThaiNLP / PyThaiNLP/pythainlp
bug: Warning: Duplicate word in word2vec file
Open
@bact is already working on this.
Since Mar 9, 2026.
bug
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 304
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 14
Description
Description
There are hundreds of warnings like this during unit test:
2023-12-11:03:40:47 WARNING [gensim.models.keyedvectors:1909] duplicate word 'ต่าง' in word2vec file, ignoring all but first
Expected results
No warning.
Current results
(partial)
2023-12-11:03:40:47 WARNING [gensim.models.keyedvectors:1909] duplicate word 'ต่าง' in word2vec file, ignoring all but first
2023-12-11:03:40:47 WARNING [gensim.models.keyedvectors:1909] duplicate word ' ' in word2vec file, ignoring all but first
...
2023-12-11:03:40:57 WARNING [gensim.models.keyedvectors:1909] duplicate word '' in word2vec file, ignoring all but first
2023-12-11:03:40:58 WARNING [gensim.models.keyedvectors:1909] duplicate word 'หยับ' in word2vec file, ignoring all but first
Steps to reproduce
Run unit test
PyThaiNLP version
dev
Python version
3.8
Operating system and version
n/a
More info
No response
Possible solution
No response
Files
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.