TestRandomChains.testRandomChainsWithLargeStrings() failures [LUCENE-7805]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
My Jenkins found a reproducing master seed, looks like FlattenGraphFilter is where the problem happens:
```
Checking out Revision 680f4d7fd378868254786107de92a894758f667c (refs/remotes/origin/master)
[...]
[junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains
[junit4] 2> TEST FAIL: useCharFilter=false text='\u0003J\u522f nwqbl uwtps ob zdyokom ){0'
[junit4] 2> Exception from random analyzer:
[junit4] 2> charfilters=
[junit4] 2> org.apache.lucene.analysis.charfilter.HTMLStripCharFilter(java.io.StringReader@3ab617ae, [])
[junit4] 2> org.apache.lucene.analysis.charfilter.HTMLStripCharFilter(org.apache.lucene.analysis.charfilter.HTMLStripCharFilter@23e3c717)
[junit4] 2> tokenizer=
[junit4] 2> org.apache.lucene.analysis.ngram.NGramTokenizer(9, 43)
[junit4] 2> filters=
[junit4] 2> org.apache.lucene.analysis.miscellaneous.CodepointCountFilter(ValidatingTokenFilter@6b4708ea term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word, 33, 44)
[junit4] 2> org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@5533fb25 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word, )
[junit4] 2> org.apache.lucene.analysis.core.FlattenGraphFilter(ValidatingTokenFilter@4ef4c44 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word)
[junit4] 2> org.apache.lucene.analysis.miscellaneous.KeepWordFilter(ValidatingTokenFilter@15baa1c7 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word, [akbnucwt, vrkwm, jtomhk, jxgmfalr])
[junit4] 2> offsetsAreCorrect=true
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestRandomChains -Dtests.method=testRandomChainsWithLargeStrings -Dtests.seed=E9460213902F2F82 -Dtests.slow=true -Dtests.locale=fi-FI -Dtests.timezone=Europe/Malta -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 0.19s J7 | TestRandomChains.testRandomChainsWithLargeStrings <<<
[junit4] > Throwable #1: java.lang.AssertionError: outputEndNode=3 vs inputTo=2
[junit4] > at __randomizedtesting.SeedInfo.seed([E9460213902F2F82:831DBD02C9610F71]:0)
[junit4] > at org.apache.lucene.analysis.core.FlattenGraphFilter.incrementToken(FlattenGraphFilter.java:335)
[junit4] > at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:67)
[junit4] > at org.apache.lucene.analysis.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:51)
[junit4] > at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:67)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:731)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:642)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:540)
[junit4] > at org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings(TestRandomChains.java:880)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {dummy=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{}, maxPointsInLeafNode=807, maxMBSortInHeap=5.007333045299232, sim=RandomSimilarity(queryNorm=true): {}, locale=fi-FI, timezone=Europe/Malta
[junit4] 2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 1.8.0_77 (64-bit)/cpus=16,threads=1,free=492062472,total=525336576
```
---
Migrated from [LUCENE-7805](https://issues.apache.org/jira/browse/LUCENE-7805) by Steven Rowe (@sarowe), updated May 24 2018
Contributor guide
Research direction
Start by running the reproduced ant command for TestRandomChains.testRandomChainsWithLargeStrings using seed E9460213902F2F82. Read TestRandomChains.java around line 880 and FlattenGraphFilter.java around line 335, then trace the outputEndNode/inputTo assertion. Done means the reproducer passes without the assertion failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100