apache / apache/lucene

DefaultICUTokenizerConfig should use the default ICU behavior for the Khmer script [LUCENE-5110]

Open
#6,174 0 comments 0 reactions 0 assignees View on GitHub
affects-version:4.0 legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Recent versions of ICU have their own implementation for the tokenization of the Khmer script. Lucene should not be overriding ICU's behavior any more.

I haven't tried the patch out, but the patch should look something like the following:

$ diff DefaultICUTokenizerConfig.java.orig DefaultICUTokenizerConfig.java
67,68d66
< private static final BreakIterator thaiBreakIterator =
< BreakIterator.getWordInstance(new ULocale("th_TH"));
71,72d68
< private static final BreakIterator khmerBreakIterator =
< readBreakIterator("Khmer.brk");
87d82
< case UScript.THAI: return (BreakIterator)thaiBreakIterator.clone();
89d83
< case UScript.KHMER: return (BreakIterator)khmerBreakIterator.clone();

and the Khmer.\* files should be removed. ICU already does script specific tokenization these days. So the Thai one should not be needed either since ICU 50.

---
Migrated from [LUCENE-5110](https://issues.apache.org/jira/browse/LUCENE-5110) by George Rhoten

Contributor guide

Open the contributing guide

Research direction

Start with DefaultICUTokenizerConfig.java and compare its script-specific BreakIterator handling with the ICU behavior described in the issue. Check the Khmer.* resources and the Thai and Khmer branches; done means the overrides and obsolete resources are removed so ICU supplies script-specific tokenization.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.