DefaultICUTokenizerConfig should use the default ICU behavior for the Khmer script [LUCENE-5110]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Recent versions of ICU have their own implementation for the tokenization of the Khmer script. Lucene should not be overriding ICU's behavior any more.
I haven't tried the patch out, but the patch should look something like the following:
$ diff DefaultICUTokenizerConfig.java.orig DefaultICUTokenizerConfig.java
67,68d66
< private static final BreakIterator thaiBreakIterator =
< BreakIterator.getWordInstance(new ULocale("th_TH"));
71,72d68
< private static final BreakIterator khmerBreakIterator =
< readBreakIterator("Khmer.brk");
87d82
< case UScript.THAI: return (BreakIterator)thaiBreakIterator.clone();
89d83
< case UScript.KHMER: return (BreakIterator)khmerBreakIterator.clone();
and the Khmer.\* files should be removed. ICU already does script specific tokenization these days. So the Thai one should not be needed either since ICU 50.
---
Migrated from [LUCENE-5110](https://issues.apache.org/jira/browse/LUCENE-5110) by George Rhoten
Contributor guide
Research direction
Start with DefaultICUTokenizerConfig.java and compare its script-specific BreakIterator handling with the ICU behavior described in the issue. Check the Khmer.* resources and the Thai and Khmer branches; done means the overrides and obsolete resources are removed so ICU supplies script-specific tokenization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100