apache / apache/lucene

TestRandomChains.testRandomChainsWithLargeStrings() failures [LUCENE-7805]

Open
#8,856 4 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

My Jenkins found a reproducing master seed, looks like FlattenGraphFilter is where the problem happens:

```
Checking out Revision 680f4d7fd378868254786107de92a894758f667c (refs/remotes/origin/master)
[...]
[junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains
[junit4] 2> TEST FAIL: useCharFilter=false text='\u0003J\u522f nwqbl uwtps ob zdyokom ){0'
[junit4] 2> Exception from random analyzer:
[junit4] 2> charfilters=
[junit4] 2> org.apache.lucene.analysis.charfilter.HTMLStripCharFilter(java.io.StringReader@3ab617ae, [])
[junit4] 2> org.apache.lucene.analysis.charfilter.HTMLStripCharFilter(org.apache.lucene.analysis.charfilter.HTMLStripCharFilter@23e3c717)
[junit4] 2> tokenizer=
[junit4] 2> org.apache.lucene.analysis.ngram.NGramTokenizer(9, 43)
[junit4] 2> filters=
[junit4] 2> org.apache.lucene.analysis.miscellaneous.CodepointCountFilter(ValidatingTokenFilter@6b4708ea term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word, 33, 44)
[junit4] 2> org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@5533fb25 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word, )
[junit4] 2> org.apache.lucene.analysis.core.FlattenGraphFilter(ValidatingTokenFilter@4ef4c44 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word)
[junit4] 2> org.apache.lucene.analysis.miscellaneous.KeepWordFilter(ValidatingTokenFilter@15baa1c7 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word, [akbnucwt, vrkwm, jtomhk, jxgmfalr])
[junit4] 2> offsetsAreCorrect=true
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestRandomChains -Dtests.method=testRandomChainsWithLargeStrings -Dtests.seed=E9460213902F2F82 -Dtests.slow=true -Dtests.locale=fi-FI -Dtests.timezone=Europe/Malta -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 0.19s J7 | TestRandomChains.testRandomChainsWithLargeStrings <<<
[junit4] > Throwable #1: java.lang.AssertionError: outputEndNode=3 vs inputTo=2
[junit4] > at __randomizedtesting.SeedInfo.seed([E9460213902F2F82:831DBD02C9610F71]:0)
[junit4] > at org.apache.lucene.analysis.core.FlattenGraphFilter.incrementToken(FlattenGraphFilter.java:335)
[junit4] > at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:67)
[junit4] > at org.apache.lucene.analysis.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:51)
[junit4] > at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:67)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:731)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:642)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:540)
[junit4] > at org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings(TestRandomChains.java:880)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {dummy=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{}, maxPointsInLeafNode=807, maxMBSortInHeap=5.007333045299232, sim=RandomSimilarity(queryNorm=true): {}, locale=fi-FI, timezone=Europe/Malta
[junit4] 2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 1.8.0_77 (64-bit)/cpus=16,threads=1,free=492062472,total=525336576
```

---
Migrated from [LUCENE-7805](https://issues.apache.org/jira/browse/LUCENE-7805) by Steven Rowe (@sarowe), updated May 24 2018

Contributor guide

Open the contributing guide

Research direction

Start by running the reproduced ant command for TestRandomChains.testRandomChainsWithLargeStrings using seed E9460213902F2F82. Read TestRandomChains.java around line 880 and FlattenGraphFilter.java around line 335, then trace the outputEndNode/inputTo assertion. Done means the reproducer passes without the assertion failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.