apache / apache/lucene

JapaneseCompletionFilter messes up offsets [LUCENE-10363]

Open
#11,399 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-label:random-chains legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

It is a tokenfilter, tries to change offsets, so of course TestRandomChains finds bugs in it:
```
NOTE: reproduce with: gradlew test --tests TestRandomChains.testRandomChainsWithLargeStrings -Dtests.seed=E233A5FAC016E02 -Dtests.nightly=true -Dtests.slow=true -Dtests.locale=en-TV -Dtests.timezone=Asia/Saigon -Dtests.asserts=true -Dtests.file.encoding=UTF-8
```

```
org.apache.lucene.analysis.tests.TestRandomChains > test suite's output saved to /home/rmuir/workspace/lucene/lucene/analysis/integration.tests/build/test-results/test_54/outputs/OUTPUT-org.apache.lucene.analysis.tests.TestRandomChains.txt, copied below:
2> stage 0: lk<[1-3] +1> p<[6-7] +1> ngtoixtmldzsjz<[10-24] +1> uoq<[25-28] +1> HANGUL<[28-28] +1> o<[29-30] +1> HANGUL<[31-31] +1> VulliPHsZzn<[32-43] +1>
2> stage 1: lk<[1-3] +1> 850000<[1-3] +0> p<[6-7] +1> 700000<[6-7] +0> ngtoixtmldzsjz<[10-24] +1> 653543<[10-24] +0> uoq<[25-28] +1> 050000<[25-28] +0> HANGUL<[28-28] +1> 565800<[28-28] +0> o<[29-30] +1> 000000<[29-30] +0> HANGUL<[31-31] +1> 565800<[31-31] +0> VulliPHsZzn<[32-43] +1> 787460<[32-43] +0>
2> stage 2: ngtoixtmldzsjz 653543<[10-24] +0> 653543<[10-24] +1> 653543 uoq<[10-28] +0> uoq<[25-28] +1> uoq 050000<[25-28] +0> 050000<[25-28] +1> 050000 HANGUL<[25-28] +0> HANGUL<[28-28] +1> HANGUL 565800<[28-28] +0> 565800<[28-28] +1> 565800 o<[28-30] +0> o<[29-30] +1> o 000000<[29-30] +0> 000000<[29-30] +1> 000000 HANGUL<[29-31] +0> HANGUL<[31-31] +1> HANGUL 565800<[31-31] +0> 565800<[31-31] +1> 565800 VulliPHsZzn<[31-43] +0> VulliPHsZzn<[32-43] +1>
2> last stage: ngtoixtmldzsjz<[10-24] +1> ngtoixtmldzsjz 653543<[10-24] +0> 653543<[10-24] +1> 653543 uoq<[10-28] +0> uoq<[25-28] +1> uoq 050000<[25-28] +1> 050000<[25-28] +1> 050000 HANGUL<[25-28] +1> HANGUL<[28-28] +1> HANGUL 565800<[28-28] +0> 565800<[28-28] +1> 565800 o<[28-30] +0> o<[29-30] +1> o 000000<[29-30] +0> 000000<[29-30] +1> 000000 HANGUL<[29-31] +0> HANGUL<[31-31] +1> HANGUL 565800<[31-31] +1> 565800<[31-31] +1> 565800 VulliPHsZzn<[31-43] +0>
2> TEST FAIL: useCharFilter=true text='[lk[-.p|) ngtoixtmldzsjz uoqao aVulliPHsZzn wxsk'
2> Exception from random analyzer:
2> charfilters=
2> org.apache.lucene.analysis.pattern.PatternReplaceCharFilter(a, , java.io.StringReader@5b3b54eb)
2> tokenizer=
2> org.apache.lucene.analysis.classic.ClassicTokenizer(org.apache.lucene.util.AttributeFactory$1@e29311e9)
2> filters=
2> org.apache.lucene.analysis.phonetic.DaitchMokotoffSoundexFilter(ValidatingTokenFilter@32a6de77 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1, true)
2> org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@3d044414 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1, q)
2> Conditional:org.apache.lucene.analysis.ja.JapaneseCompletionFilter(OneTimeWrapper@435207ec term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,reading=null,reading (en)=null,pronunciation=null,pronunciation (en)=null, INDEX)
> java.lang.IllegalStateException: last stage: inconsistent endOffset at pos=19: 31 vs 43; token=565800 VulliPHsZzn
```

---
Migrated from [LUCENE-10363](https://issues.apache.org/jira/browse/LUCENE-10363) by Robert Muir (@rmuir)

Contributor guide

Open the contributing guide

Research direction

Start with JapaneseCompletionFilter and reproduce the failure using TestRandomChains.testRandomChainsWithLargeStrings and the supplied Gradle command and seed. Trace how the filter handles token offsets in the reported random analyzer chain. Done means the reproducer passes without an inconsistent endOffset exception and offset validation remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.