TestICUTokenizerCJK failure [LUCENE-8222]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
This reproduces for me:
```java
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestICUTokenizerCJK -Dtests.method=testRandomHugeStrings -Dtests.seed=2C1F4414ECB02FE4 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=pl-PL -Dtests.timezone=Europe/Athens -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 0.57s | TestICUTokenizerCJK.testRandomHugeStrings <<<
[junit4] > Throwable #1: org.junit.ComparisonFailure: term 128 expected:<ー[]> but was:<ー[デ]>
[junit4] > at __randomizedtesting.SeedInfo.seed([2C1F4414ECB02FE4:B43C23D7B2C693AC]:0)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:201)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:325)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:329)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:869)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:668)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:566)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:479)
[junit4] > at org.apache.lucene.analysis.icu.segmentation.TestICUTokenizerCJK.testRandomHugeStrings(TestICUTokenizerCJK.java:107)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:564)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {dummy=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{}, maxPointsInLeafNode=1800, maxMBSortInHeap=7.290162896982681, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@3f70f3d9), locale=pl-PL, timezone=Europe/Athens
[junit4] 2> NOTE: Mac OS X 10.13.3 x86_64/Oracle Corporation 9.0.1 (64-bit)/cpus=8,threads=1,free=150530048,total=268435456
```
---
Migrated from [LUCENE-8222](https://issues.apache.org/jira/browse/LUCENE-8222) by Alan Woodward (@romseygeek), updated Dec 18 2019
Contributor guide
Research direction
Start by running the reproduced ant command for TestICUTokenizerCJK.testRandomHugeStrings and inspect org/apache/lucene/analysis/icu/segmentation/TestICUTokenizerCJK.java:107. Trace the reported token mismatch through the tokenizer and related test helpers. Done means the failure is understood and the test passes consistently with the intended tokenization behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100