apache / apache/lucene

JapaneseIterationMarkCharFilter: TestRandomChains fails with incorrect offsets or causes IndexOutOfBounds [LUCENE-10358]

Open
#11,394 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-label:random-chains legacy-jira-priority:Major module:analysis type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Failures seen:
```
$ gradlew :lucene:analysis:integration.tests:test --tests TestRandomChains.testRandomChainsWithLargeStrings -Dtests.seed=AA632771CC823702 -Dtests.slow=true -Dtests.locale=fr-MF -Dtests.timezone=America/Panama -Dtests.asserts=true -Dtests.file.encoding=UTF-8
org.apache.lucene.analysis.tests.TestRandomChains > test suite's output saved to C:\Users\Uwe Schindler\Projects\lucene\lucene\lucene\analysis\integration.tests\build\test-results\test\outputs\OUTPUT-org.apache.lucene.analysis.tests.TestRandomChains.txt, copied below:
2> stage 0: ­ÉÆû<[0-2] +1> ­ÉÆä<[4-6] +1> ppkarrpf<[7-14] +1> 1<[16-17] +1> 5<[18-19] +1>
2> stage 1: ­ÉÆû<[0-2] +1> ­ÉÆä<[4-6] +1> 000000<[4-6] +0> ppkarrpf<[7-14] +1> 759700<[7-14] +0> 1<[16-17] +1> 5<[18-19] +1> 000000<[18-19] +0>
2> stage 2: ­ÉÆû<[0-2] +1> ­ÉÆä<[4-6] +1> 000000<[4-6] +0> ppkarrpf<[7-14] +1> 759700<[7-14] +0> 1<[16-17] +1> 000000<[18-19] +0>
2> TEST FAIL: useCharFilter=true text='\ud801\udc96\ud801\udcaa\ud801\udc84 ppkarpf {1,5}g?)u em mbm hbil'
2> Exception from random analyzer:
2> charfilters=
2> org.apache.lucene.analysis.ja.JapaneseIterationMarkCharFilter(java.io.StringReader@105e6aa7, true, false)
2> tokenizer=
2> org.apache.lucene.analysis.th.ThaiTokenizer()
2> filters=
2> Conditional:org.apache.lucene.analysis.phonetic.DaitchMokotoffSoundexFilter(OneTimeWrapper@79889b7f term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1, true)
2> org.apache.lucene.analysis.ja.JapaneseNumberFilter(ValidatingTokenFilter@53a9e96c term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,keyword=false)
2> org.apache.lucene.analysis.miscellaneous.StemmerOverrideFilter(ValidatingTokenFilter@6cb4578d term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,keyword=false, org.apache.lucene.analysis.miscellaneous.StemmerOverrideFilter$StemmerOverrideMap@51fc8124)
> java.lang.IllegalStateException: stage 2: inconsistent startOffset at pos=3: 16 vs 18; token=000000
> at __randomizedtesting.SeedInfo.seed([AA632771CC823702:C038986095CC17F1]:0)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:138)
> at org.apache.lucene.analysis.common@10.0.0-SNAPSHOT/org.apache.lucene.analysis.miscellaneous.StemmerOverrideFilter.incrementToken(StemmerOverrideFilter.java:67)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:81)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:1130)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:1028)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:922)
> at org.apache.lucene.analysis.tests@10.0.0-SNAPSHOT/org.apache.lucene.analysis.tests.TestRandomChains.testRandomChainsWithLargeStrings(TestRandomChains.java:943)
```

and also:

```
$ gradlew :lucene:analysis:integration.tests:test --tests TestRandomChains.testRandomChains -Dtests.seed=3A0D0E91E0CA5BFC -Dtests.slow=true -Dtests.locale=nmg-CM -Dtests.timezone=Antarctica/Vostok -Dtests.asserts=true -Dtests.file.encoding=UTF-8
org.apache.lucene.analysis.tests.TestRandomChains > test suite's output saved to C:\Users\Uwe Schindler\Projects\lucene\lucene\lucene\analysis\integration.tests\build\test-results\test_17\outputs\OUTPUT-org.apache.lucene.analysis.tests.TestRandomChains.txt, copied below:
2> TEST FAIL: useCharFilter=false text=''
2> Exception from random analyzer:
2> charfilters=
2> org.apache.lucene.analysis.ja.JapaneseIterationMarkCharFilter(java.io.StringReader@7ee7c045)
2> org.apache.lucene.analysis.charfilter.HTMLStripCharFilter(org.apache.lucene.analysis.ja.JapaneseIterationMarkCharFilter@66bba53d, [])
2> tokenizer=
2> org.apache.lucene.analysis.core.KeywordTokenizer(27)
2> filters=
2> org.apache.lucene.analysis.ja.JapaneseKatakanaStemFilter(ValidatingTokenFilter@1a8623a7 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,keyword=false, -21)
> java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 32
> at __randomizedtesting.SeedInfo.seed([3A0D0E91E0CA5BFC:7EC27F0A7D8463C]:0)
> at org.apache.lucene.analysis.kuromoji@10.0.0-SNAPSHOT/org.apache.lucene.analysis.ja.JapaneseKatakanaStemFilter.stem(JapaneseKatakanaStemFilter.java:76)
> at org.apache.lucene.analysis.kuromoji@10.0.0-SNAPSHOT/org.apache.lucene.analysis.ja.JapaneseKatakanaStemFilter.incrementToken(JapaneseKatakanaStemFilter.java:59)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:81)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:1130)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:1028)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:922)
> at org.apache.lucene.analysis.tests@10.0.0-SNAPSHOT/org.apache.lucene.analysis.tests.TestRandomChains.testRandomChains(TestRandomChains.java:911)
```

---
Migrated from [LUCENE-10358](https://issues.apache.org/jira/browse/LUCENE-10358) by Uwe Schindler (@uschindler)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the two reported TestRandomChains failures with their supplied Gradle commands and seeds. Read JapaneseIterationMarkCharFilter and JapaneseKatakanaStemFilter.java around line 76, then trace the failing analysis chains; done means the random-chain tests no longer report inconsistent offsets or ArrayIndexOutOfBoundsException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.