Better support for overlapping scripts (eg. CJK) in the segmenter
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 13
- Forks
- 8
- Avg merge
- 3h 57m
- Merged PRs (30d)
- 7
Description
In the segmenter within a particular script merging is only performed for codepoints that are unique to that script. We need to extend the segmenter to be able to handle segmentations when scripts with large overlaps are provided. The common example of this CJK. The segmenter documentation discusses this in more detail and proposes a solution, see the "Multiple Frequency Data Sets and Merge Groups" section.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the segmenter documentation's "Multiple Frequency Data Sets and Merge Groups" section, then locate the segmenter implementation it describes. Trace how merging currently handles codepoints unique to one script and determine how overlapping scripts such as CJK should be segmented; the work is done when those overlapping-script cases are supported as specified by the documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100