apache / apache/lucene

KoreanTokenizer: TestRandomChains fails with incorrect offsets [LUCENE-10359]

Open
#11,395 6 comments 0 reactions 0 assignees View on GitHub
legacy-jira-label:random-chains legacy-jira-priority:Major module:analysis type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

It looks like KoreanTokenizer is causing this (NORI), but Kuromoji may be affected in the same way:

```
org.apache.lucene.analysis.tests.TestRandomChains > test suite's output saved to C:\Users\Uwe Schindler\Projects\lucene\lucene\lucene\analysis\integration.tests\build\test-results\test\outputs\OUTPUT-org.apache.lucene.analysis.tests.TestRandomChains.txt, copied below:
2> stage 0: e<[2-3] +1> ek<[4-6] +1> oy<[8-10] +1> 1<[11-12] +1> zzkuxp<[13-19] +1>
2> stage 1: e<[2-3] +1> ek<[4-6] +1> oy<[8-10] +1> 1<[11-12] +1> zzkuxp<[13-19] +1>
2> stage 2: e<[2-3] +1> e ek<[2-6] +0> ek<[4-6] +1> ek oy<[4-10] +0> oy<[8-10] +1> oy 1<[8-12] +0> 1<[11-12] +1> 1 zzkuxp<[11-19] +0>
2> stage 3: e<[2-3] +1> e ek<[2-6] +0> ek<[4-6] +1> ek oy<[4-10] +0> oy<[8-10] +1> oy 1<[8-12] +0> 1<[11-12] +1> 1 zzkuxp<[11-19] +0>
2> last stage: e<[2-3] +1> e ek<[2-6] +0> ek<[4-6] +1> ek oy<[4-10] +0> oy<[8-10] +1> oy 1<[8-12] +0> 1 zzkuxp<[11-19] +0>
2> TEST FAIL: useCharFilter=false text='?.e|ek|]oy{1 zzkuxp ZyzzV ycuqjnv axtpppvk \u233b\u23c8\u2314\u232e\u236e\u238d\u235e x d \"

'
2> Exception from random analyzer:
2> charfilters=
2> org.apache.lucene.analysis.pattern.PatternReplaceCharFilter(a, ifywufhi, java.io.StringReader@48586999)
2> org.apache.lucene.analysis.charfilter.MappingCharFilter(org.apache.lucene.analysis.charfilter.NormalizeCharMap@65036838, org.apache.lucene.analysis.pattern.PatternReplaceCharFilter@11d4ba35)
2> tokenizer=
2> org.apache.lucene.analysis.ko.KoreanTokenizer()
2> filters=
2> org.apache.lucene.analysis.en.KStemFilter(ValidatingTokenFilter@595d7938 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,posType=null,leftPOS=null,rightPOS=null,morphemes=null,reading=null,keyword=false)
2> org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@13d08b48 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,posType=null,leftPOS=null,rightPOS=null,morphemes=null,reading=null,keyword=false, u)
2> org.apache.lucene.analysis.util.ElisionFilter(ValidatingTokenFilter@6396b917 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,posType=null,leftPOS=null,rightPOS=null,morphemes=null,reading=null,keyword=false, [fh, hiiwwxyyd, fcpodqor, qogvhmywr, l, icad])
2> Conditional:org.apache.lucene.analysis.ko.KoreanNumberFilter(OneTimeWrapper@5f0558f6 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,posType=null,leftPOS=null,rightPOS=null,morphemes=null,reading=null,keyword=false)
> java.lang.IllegalStateException: last stage: inconsistent startOffset at pos=2: 8 vs 11; token=1 zzkuxp
> at __randomizedtesting.SeedInfo.seed([E4552C7844FC2DA3:8E0E93691DB20D50]:0)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:138)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:1130)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:1028)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:922)
> at org.apache.lucene.analysis.tests@10.0.0-SNAPSHOT/org.apache.lucene.analysis.tests.TestRandomChains.testRandomChainsWithLargeStrings(TestRandomChains.java:943)
```

---
Migrated from [LUCENE-10359](https://issues.apache.org/jira/browse/LUCENE-10359) by Uwe Schindler (@uschindler), updated Apr 09 2022

Contributor guide

Open the contributing guide

Research direction

Start with TestRandomChains.java at line 943 and reproduce the failure involving KoreanTokenizer, then inspect the reported offset validation in ValidatingTokenFilter. Compare the stages in the saved output and check whether Kuromoji shows the same behavior. Done means the random-chain test no longer reports inconsistent startOffset values for this analyzer chain.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.