org.apache.lucene.analysis.tests.TestRandomChains.testRandomChains test failure
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Haven't finished bisection to figure out the origin of the bug.
```
org.apache.lucene.analysis.tests.TestRandomChains > testRandomChains FAILED
java.lang.IllegalStateException: last stage: inconsistent endOffset at pos=1: 8 vs 11; token=ڎbdb觸 Ǧ
at __randomizedtesting.SeedInfo.seed([3AEBBB09EA70DCE:3E4F92D1D9B5100E]:0)
at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:164)
at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:1289)
at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:1187)
at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:1081)
at org.apache.lucene.analysis.tests@10.0.0-SNAPSHOT/org.apache.lucene.analysis.tests.TestRandomChains.testRandomChains(TestRandomChains.java:945)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
```
```
2> stage 0: ڎ<[1-2] +1> d<[4-5] +1> 觸<[7-8] +1> Ǧ<[10-11] +1>
2> stage 1: ڎ<[1-2] +1> d<[4-5] +1> 觸<[7-8] +1> Ǧ<[10-11] +1>
2> stage 2: ڎ<[1-2] +1> ڎ d<[1-5] +0> d<[4-5] +1> d 觸<[4-8] +0> 觸<[7-8] +1> 觸 Ǧ<[7-11] +0>
2> last stage: ڎbdb觸<[1-8] +1> ڎbdb觸 Ǧ<[1-11] +0>
2> TEST FAIL: useCharFilter=true text='\u0003\ufb87\udacd\uddf6d\uf1f4\u02e6\u89f8\uda06\udfd2\u01e6'
2> Exception from random analyzer:
2> charfilters=
2> org.apache.lucene.analysis.icu.ICUNormalizer2CharFilter(java.io.StringReader@24c09603, com.ibm.icu.impl.Norm2AllModes$ComposeNormalizer2@43b7067e)
2> tokenizer=
2> org.apache.lucene.analysis.th.ThaiTokenizer(org.apache.lucene.util.AttributeFactory$1@a0c154ba)
2> filters=
2> org.apache.lucene.analysis.no.NorwegianMinimalStemFilter(ValidatingTokenFilter@215f24e9 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,flags=0,payload=null,keyword=false)
2> org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@47f9add2 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,flags=0,payload=null,keyword=false, fg)
2> org.apache.lucene.analysis.shingle.FixedShingleFilter(ValidatingTokenFilter@71cd79a term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,flags=0,payload=null,keyword=false, 3, b, )
> java.lang.IllegalStateException: last stage: inconsistent endOffset at pos=1: 8 vs 11; token=ڎbdb觸 Ǧ
```
### Gradle command to reproduce
```
./gradlew test --tests TestRandomChains.testRandomChains -Dtests.seed=3AEBBB09EA70DCE -Dtests.nightly=true -Dtests.locale=en-CA -Dtests.timezone=America/Cordoba -Dtests.asserts=true -Dtests.file.encoding=UTF-8
```
Contributor guide
Research direction
Run the supplied Gradle command for TestRandomChains.testRandomChains with the recorded seed. Start at TestRandomChains.java:945 and inspect ValidatingTokenFilter.java:164, following the reported ThaiTokenizer, ShingleFilter, and FixedShingleFilter chain. Done means the inconsistent endOffset failure is understood and the reproducing test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100