apache / apache/lucene

org.apache.lucene.analysis.tests.TestRandomChains.testRandomChains test failure

Open
#13,271 15 comments 0 reactions 0 assignees View on GitHub
type:test
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Haven't finished bisection to figure out the origin of the bug.

```
org.apache.lucene.analysis.tests.TestRandomChains > testRandomChains FAILED
java.lang.IllegalStateException: last stage: inconsistent endOffset at pos=1: 8 vs 11; token=ڎbdb觸 Ǧ
at __randomizedtesting.SeedInfo.seed([3AEBBB09EA70DCE:3E4F92D1D9B5100E]:0)
at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:164)
at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:1289)
at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:1187)
at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:1081)
at org.apache.lucene.analysis.tests@10.0.0-SNAPSHOT/org.apache.lucene.analysis.tests.TestRandomChains.testRandomChains(TestRandomChains.java:945)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
```

```
2> stage 0: ڎ<[1-2] +1> d<[4-5] +1> 觸<[7-8] +1> Ǧ<[10-11] +1>
2> stage 1: ڎ<[1-2] +1> d<[4-5] +1> 觸<[7-8] +1> Ǧ<[10-11] +1>
2> stage 2: ڎ<[1-2] +1> ڎ d<[1-5] +0> d<[4-5] +1> d 觸<[4-8] +0> 觸<[7-8] +1> 觸 Ǧ<[7-11] +0>
2> last stage: ڎbdb觸<[1-8] +1> ڎbdb觸 Ǧ<[1-11] +0>
2> TEST FAIL: useCharFilter=true text='\u0003\ufb87\udacd\uddf6d\uf1f4\u02e6\u89f8\uda06\udfd2\u01e6'
2> Exception from random analyzer:
2> charfilters=
2> org.apache.lucene.analysis.icu.ICUNormalizer2CharFilter(java.io.StringReader@24c09603, com.ibm.icu.impl.Norm2AllModes$ComposeNormalizer2@43b7067e)
2> tokenizer=
2> org.apache.lucene.analysis.th.ThaiTokenizer(org.apache.lucene.util.AttributeFactory$1@a0c154ba)
2> filters=
2> org.apache.lucene.analysis.no.NorwegianMinimalStemFilter(ValidatingTokenFilter@215f24e9 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,flags=0,payload=null,keyword=false)
2> org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@47f9add2 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,flags=0,payload=null,keyword=false, fg)
2> org.apache.lucene.analysis.shingle.FixedShingleFilter(ValidatingTokenFilter@71cd79a term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,flags=0,payload=null,keyword=false, 3, b, )
> java.lang.IllegalStateException: last stage: inconsistent endOffset at pos=1: 8 vs 11; token=ڎbdb觸 Ǧ
```

### Gradle command to reproduce

```
./gradlew test --tests TestRandomChains.testRandomChains -Dtests.seed=3AEBBB09EA70DCE -Dtests.nightly=true -Dtests.locale=en-CA -Dtests.timezone=America/Cordoba -Dtests.asserts=true -Dtests.file.encoding=UTF-8
```

Contributor guide

Open the contributing guide

Research direction

Run the supplied Gradle command for TestRandomChains.testRandomChains with the recorded seed. Start at TestRandomChains.java:945 and inspect ValidatingTokenFilter.java:164, following the reported ThaiTokenizer, ShingleFilter, and FixedShingleFilter chain. Done means the inconsistent endOffset failure is understood and the reproducing test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.