End offset for compatibility characters is not incremented with ICUNormalizer2CharFilter
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
This issue comes from https://github.com/elastic/elasticsearch/issues/50008.
When tokenizing combining characters (ex. `㋀`) after applying the char filter `icu_normalizer`, end offset of combining character is not incremented correctly.
The test which I added in [TestICUNormalizer2CharFilter]( https://github.com/apache/lucene/blob/main/lucene/analysis/icu/src/test/org/apache/lucene/analysis/icu/TestICUNormalizer2CharFilter.java) failed.
```java
public void testTokenStreamCombiningCharacter() throws IOException {
String input = "日日㋀日"; // ㋀ is the combining character
CharFilter reader =
new ICUNormalizer2CharFilter(
new StringReader(input),
Normalizer2.getInstance(null, "nfkc_cf", Normalizer2.Mode.COMPOSE));
Tokenizer tokenStream =
new ICUTokenizer(newAttributeFactory(), new DefaultICUTokenizerConfig(false, true));
tokenStream.setReader(reader);
assertTokenStreamContents(
tokenStream,
new String[] {"日", "日", "1", "月", "日"},
new int[] {0, 1, 2, 3, 4}, // test pass if changed to {0, 1, 2, 2, 3}
new int[] {1, 2, 3, 4, 5}, // test pass if changed to {1, 2, 2, 3, 4} (end offset for the word `1` is not incremented)
input.length());
}
```
```
$ ./gradlew test --tests org.apache.lucene.analysis.icu.TestICUNormalizer2CharFilter.testTokenStreamCombiningCharacter
org.apache.lucene.analysis.icu.TestICUNormalizer2CharFilter > testTokenStreamCombiningCharacter FAILED
java.lang.AssertionError: endOffset 2 term=1 expected:<3> but was:<2>
```
### Version and environment details
- macOS 12.3.1
- openjdk 17.0.5
Contributor guide
Research direction
Start by running the named Gradle test, org.apache.lucene.analysis.icu.TestICUNormalizer2CharFilter.testTokenStreamCombiningCharacter. Read TestICUNormalizer2CharFilter and the ICUNormalizer2CharFilter path involved in tokenization, then verify that the combining-character case reports the expected end offsets and the test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100