google / google/budoux

[quality] Long words in zh-hans model (20198 suggested changes)

Open
#874 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.8k
Forks
45
Avg merge
22h 43m
Merged PRs (30d)
32

Description

Thank you for making budoux! I've been using it actively for Chinese (Traditional), Chinese (Simplified), Japanese, and Thai. It's very fast, and I really appreciate your work on it!

Input: [UNv1.0.en-zh.zh](https://www.un.org/dgacm/en/content/uncorpus/Download)

Process:
`cat "/Users/peter/Downloads/budoux-main/UNv1.0.en-zh.zh" | python3 budoux/main.py -m 'budoux/models/zh-hans.json' > "/Users/peter/Downloads/budoux-main/UNv1.0.en-zh.zhSpaced.txt"`

Expected output (sample):

> 基 皮亚 克 土著马 赛 群体 争取 生存 计划
> 释放 利比亚国民 阿卜杜勒 巴塞特
> 波斯尼亚 - 克罗地亚 - 塞尔维亚
> ( 阿波 斯托 洛斯安 德 列 亚斯 角)

Expected output (full):
[UNv1.0.en-zh.zhSpacedWordsOver5CharactersSpaced.txt](https://github.com/user-attachments/files/18374218/UNv1.0.en-zh.zhSpacedWordsOver5CharactersSpaced.txt)

Expected output is built using a development copy of [https://pingtype.github.io/](https://pingtype.github.io/)

Actual output (sample):

> 基皮亚克土著马赛群体争取生存计划
> 释放利比亚国民阿卜杜勒巴塞特
> 波斯尼亚-克罗地亚-塞尔维亚
> (阿波斯托洛斯安德列亚斯角)

Actual output (full)
[UNv1.0.en-zh.zhSpacedWordsOver5Characters.txt](https://github.com/user-attachments/files/18374879/UNv1.0.en-zh.zhSpacedWordsOver5Characters.txt)

Please message me if you have any more questions, and I'd be happy to advise. I also have more data for long words (over 5 characters) in Japanese, Chinese (Traditional), and Thai - please comment here or email me when you're working on this issue, and I can collaborate with you more :)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.