apache / apache/lucene

End offset for compatibility characters is not incremented with ICUNormalizer2CharFilter

Open
#11,976 8 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

This issue comes from https://github.com/elastic/elasticsearch/issues/50008.
When tokenizing combining characters (ex. `㋀`) after applying the char filter `icu_normalizer`, end offset of combining character is not incremented correctly.

The test which I added in [TestICUNormalizer2CharFilter]( https://github.com/apache/lucene/blob/main/lucene/analysis/icu/src/test/org/apache/lucene/analysis/icu/TestICUNormalizer2CharFilter.java) failed.
```java
public void testTokenStreamCombiningCharacter() throws IOException {
String input = "日日㋀日"; // ㋀ is the combining character
CharFilter reader =
new ICUNormalizer2CharFilter(
new StringReader(input),
Normalizer2.getInstance(null, "nfkc_cf", Normalizer2.Mode.COMPOSE));

Tokenizer tokenStream =
new ICUTokenizer(newAttributeFactory(), new DefaultICUTokenizerConfig(false, true));
tokenStream.setReader(reader);

assertTokenStreamContents(
tokenStream,
new String[] {"日", "日", "1", "月", "日"},
new int[] {0, 1, 2, 3, 4}, // test pass if changed to {0, 1, 2, 2, 3}
new int[] {1, 2, 3, 4, 5}, // test pass if changed to {1, 2, 2, 3, 4} (end offset for the word `1` is not incremented)
input.length());
}
```
```
$ ./gradlew test --tests org.apache.lucene.analysis.icu.TestICUNormalizer2CharFilter.testTokenStreamCombiningCharacter
org.apache.lucene.analysis.icu.TestICUNormalizer2CharFilter > testTokenStreamCombiningCharacter FAILED
java.lang.AssertionError: endOffset 2 term=1 expected:<3> but was:<2>
```

### Version and environment details

- macOS 12.3.1
- openjdk 17.0.5

Contributor guide

Open the contributing guide

Research direction

Start by running the named Gradle test, org.apache.lucene.analysis.icu.TestICUNormalizer2CharFilter.testTokenStreamCombiningCharacter. Read TestICUNormalizer2CharFilter and the ICUNormalizer2CharFilter path involved in tokenization, then verify that the combining-character case reports the expected end offsets and the test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.