sillsdev / sillsdev/machine.py
Chinese text is not being tokenized properly
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 12
- Forks
- 3
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 5
Description
We currently have pretranslations generated that look like:
...
"translation": "声音已经说出来了,耶稣就独自一人了.他们在那些日子里什么都没告诉任何人.",
"translationTokens": [
"声音已经说出来了,耶稣就独自一人了.他们在那些日子里什么都没告诉任何人",
"."
],
...
I assume that this is because we are using the LatinWordTokenizer for translation alignment. This is likely happening for some other scripts as well. We should evaluate how many projects this affects and consider using another tokenizer (or dynamically choosing a tokenizer). Unless I'm missing something, this would make the marker placement feature basically unavailable for those translating into scripts that the LatinWordTokenizer does not tokenize properly.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names the LatinWordTokenizer and translation alignment but no files or tests. Start by locating LatinWordTokenizer and the alignment tokenization entry point, then measure which projects or scripts are affected and compare suitable tokenizer choices. Done means Chinese text produces usable token boundaries for marker placement, with coverage for any additional affected scripts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100