[quality] Long words in zh-hans model (20198 suggested changes)
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 45
- Avg merge
- 22h 43m
- Merged PRs (30d)
- 32
Description
Thank you for making budoux! I've been using it actively for Chinese (Traditional), Chinese (Simplified), Japanese, and Thai. It's very fast, and I really appreciate your work on it!
Input: [UNv1.0.en-zh.zh](https://www.un.org/dgacm/en/content/uncorpus/Download)
Process:
`cat "/Users/peter/Downloads/budoux-main/UNv1.0.en-zh.zh" | python3 budoux/main.py -m 'budoux/models/zh-hans.json' > "/Users/peter/Downloads/budoux-main/UNv1.0.en-zh.zhSpaced.txt"`
Expected output (sample):
> 基 皮亚 克 土著马 赛 群体 争取 生存 计划
> 释放 利比亚国民 阿卜杜勒 巴塞特
> 波斯尼亚 - 克罗地亚 - 塞尔维亚
> ( 阿波 斯托 洛斯安 德 列 亚斯 角)
Expected output (full):
[UNv1.0.en-zh.zhSpacedWordsOver5CharactersSpaced.txt](https://github.com/user-attachments/files/18374218/UNv1.0.en-zh.zhSpacedWordsOver5CharactersSpaced.txt)
Expected output is built using a development copy of [https://pingtype.github.io/](https://pingtype.github.io/)
Actual output (sample):
> 基皮亚克土著马赛群体争取生存计划
> 释放利比亚国民阿卜杜勒巴塞特
> 波斯尼亚-克罗地亚-塞尔维亚
> (阿波斯托洛斯安德列亚斯角)
Actual output (full)
[UNv1.0.en-zh.zhSpacedWordsOver5Characters.txt](https://github.com/user-attachments/files/18374879/UNv1.0.en-zh.zhSpacedWordsOver5Characters.txt)
Please message me if you have any more questions, and I'd be happy to advise. I also have more data for long words (over 5 characters) in Japanese, Chinese (Traditional), and Thai - please comment here or email me when you're working on this issue, and I can collaborate with you more :)
Contributor guide
Assessment
This issue has not been assessed yet.