apache / apache/lucene

Test2BPostingsBytes org.apache.lucene.index.CorruptIndexException: docs out of order (490879719 <= 490879719 ) [LUCENE-8925]

Open
#9,968 3 comments 0 reactions 0 assignees View on GitHub
affects-version:8.1.1 legacy-jira-label:test-failure legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

8x branch at commit 081e2ef2c05e017e87a2aef2a4f55067fbba5cb4

while running `ant -Dtests.filter=(`@monster` or `@slow`) and not(`@awaitsfix`) -Dtests.heapsize=4G -Dtests.jvms=64 test`

```
2> NOTE: reproduce with: ant test -Dtestcase=Test2BPostingsBytes -Dtests.method=test -Dtests.seed=1C14F78FC0AF1835 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=fr
-Dtests.timezone=SystemV/AST4ADT -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[23:54:00.627] ERROR 111s J52 | Test2BPostingsBytes.test <<<
> Throwable #1: org.apache.lucene.index.CorruptIndexException: docs out of order (490879719 <= 490879719 ) (resource=MockIndexOutputWrapper(FSIndexOutput(path="/home/danielgb
/lucene-solr/lucene/build/core/test/J52/temp/lucene.index.Test2BPostingsBytes_1C14F78FC0AF1835-001/2BPostingsBytes3-001/_0_Lucene50_0.doc")))
> at __randomizedtesting.SeedInfo.seed([1C14F78FC0AF1835:9440C8556E5375CD]:0)
> at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.startDoc(Lucene50PostingsWriter.java:236)
> at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:148)
> at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865)
> at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
> at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169)
> at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:245)
> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:140)
> at org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:2988)
> at org.apache.lucene.util.TestUtil.addIndexesSlowly(TestUtil.java:990)
> at org.apache.lucene.index.Test2BPostingsBytes.test(Test2BPostingsBytes.java:127)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
> at java.lang.reflect.Method.invoke(Method.java:508)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at java.lang.Thread.run(Thread.java:818)
2> NOTE: leaving temporary files on disk at: /home/danielgb/lucene-solr/lucene/build/core/test/J52/temp/lucene.index.Test2BPostingsBytes_1C14F78FC0AF1835-001
2> NOTE: test params are: codec=Lucene80, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@c792b533), locale=fr, timezone=SystemV/AST4ADT
2> NOTE: Linux 3.10.0-957.21.3.el7.ppc64le ppc64le/IBM Corporation 1.8.0_211 (64-bit)/cpus=64,threads=1,free=88221008,total=422117376
2> NOTE: All tests run in this JVM: [TestTopDocsCollector, Test2BPostingsBytes]
```

---
Migrated from [LUCENE-8925](https://issues.apache.org/jira/browse/LUCENE-8925) by Daniel Black, updated Jul 19 2019
Environment:
```
RHEL-7.3 (ppc64le - Power9)

kernel 3.10.0-957.21.3.el7.ppc64le

48G vm, 64 core

java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 8.0.5.37 - pxl6480sr5fp37-20190618_01(SR5 FP37))
IBM J9 VM (build 2.9, JRE 1.8.0 Linux ppc64le-64-Bit Compressed References 20190617_419755 (JIT enabled, AOT enabled)
OpenJ9 - 354b31d
OMR - 0437c69
IBM - 4972efe)
JCL - 20190606_01 based on Oracle jdk8u211-b25
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the provided ant command, then start at lucene/core/src/test/org/apache/lucene/index/Test2BPostingsBytes.java, especially test at line 127, and follow the stack into Lucene50PostingsWriter.startDoc. Done means the reproduced test no longer throws CorruptIndexException for the reported seed and environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.