apache / apache/lucene

TestIndexWriterExceptions.testNoLostDeletesOrUpdates() failure [LUCENE-7724]

Open
#8,775 1 comment 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Non-reproducing branch_6x seed from my Jenkins:

```
Checking out Revision bce1417fceeed2054f16565e96dc49268c1b2ea1 (refs/remotes/origin/branch_6x)
[...]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestIndexWriterExceptions -Dtests.method=testNoLostDeletesOrUpdates -Dtests.seed=611686EB379EF662 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=en-US -Dtests.timezone=America/Cordoba -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 9.90s J6 | TestIndexWriterExceptions.testNoLostDeletesOrUpdates <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=343, name=Lucene Merge Thread #21, state=RUNNABLE, group=TGRP-TestIndexWriterExceptions]
[junit4] > Caused by: org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
[junit4] > at __randomizedtesting.SeedInfo.seed([611686EB379EF662]:0)
[junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
[junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
[junit4] > Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
[junit4] > at org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:345)
[junit4] > at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:696)
[junit4] > at org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:691)
[junit4] > at org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:4964)
[junit4] > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4455)
[junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3940)
[junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
[junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
[junit4] > Caused by: org.apache.lucene.store.MockDirectoryWrapper$FakeIOException
[junit4] > at org.apache.lucene.index.TestIndexWriterExceptions$11.eval(TestIndexWriterExceptions.java:1875)
[junit4] > at org.apache.lucene.store.MockDirectoryWrapper.maybeThrowDeterministicException(MockDirectoryWrapper.java:1022)
[junit4] > at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:144)
[junit4] > at org.apache.lucene.store.MockIndexOutputWrapper.writeByte(MockIndexOutputWrapper.java:126)
[junit4] > at org.apache.lucene.store.DataOutput.writeInt(DataOutput.java:70)
[junit4] > at org.apache.lucene.codecs.CodecUtil.writeHeader(CodecUtil.java:90)
[junit4] > at org.apache.lucene.codecs.CodecUtil.writeIndexHeader(CodecUtil.java:133)
[junit4] > at org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.(Lucene54DocValuesConsumer.java:74)
[junit4] > at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsConsumer(Lucene54DocValuesFormat.java:108)
[junit4] > at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:213)
[junit4] > at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:111)
[junit4] > at org.apache.lucene.index.ReadersAndUpdates.handleNumericDVUpdates(ReadersAndUpdates.java:328)
[junit4] > at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:521)
[junit4] > at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:272)
[junit4] > at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:4119)
[junit4] > at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:4077)
[junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
[junit4] > ... 2 moreThrowable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=344, name=Lucene Merge Thread #22, state=RUNNABLE, group=TGRP-TestIndexWriterExceptions]
[junit4] > Caused by: org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.store.MockDirectoryWrapper$FakeIOException
[junit4] > at __randomizedtesting.SeedInfo.seed([611686EB379EF662]:0)
[junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
[junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
[junit4] > Caused by: org.apache.lucene.store.MockDirectoryWrapper$FakeIOException
[junit4] > at org.apache.lucene.index.TestIndexWriterExceptions$11.eval(TestIndexWriterExceptions.java:1875)
[junit4] > at org.apache.lucene.store.MockDirectoryWrapper.maybeThrowDeterministicException(MockDirectoryWrapper.java:1022)
[junit4] > at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:144)
[junit4] > at org.apache.lucene.store.MockIndexOutputWrapper.writeByte(MockIndexOutputWrapper.java:126)
[junit4] > at org.apache.lucene.store.DataOutput.writeInt(DataOutput.java:70)
[junit4] > at org.apache.lucene.codecs.CodecUtil.writeHeader(CodecUtil.java:90)
[junit4] > at org.apache.lucene.codecs.CodecUtil.writeIndexHeader(CodecUtil.java:133)
[junit4] > at org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.(Lucene54DocValuesConsumer.java:74)
[junit4] > at org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsConsumer(Lucene54DocValuesFormat.java:108)
[junit4] > at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:213)
[junit4] > at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:111)
[junit4] > at org.apache.lucene.index.ReadersAndUpdates.handleNumericDVUpdates(ReadersAndUpdates.java:328)
[junit4] > at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:521)
[junit4] > at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:272)
[junit4] > at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:4119)
[junit4] > at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:4077)
[junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
[junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
[junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=false,coord=no): {content6=DFR I(F)BZ(0.3), field=DFR I(n)BZ(0.3), content4=DFR I(F)L1, contents=DFR I(n)1, content2=DFR I(ne)Z(0.3), content1=IB LL-L1, id=DFR I(n)L3(800.0), content=DFR I(F)1}, locale=en-US, timezone=America/Cordoba
[junit4] 2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 1.8.0_77 (64-bit)/cpus=16,threads=1,free=421632952,total=456654848
```

---
Migrated from [LUCENE-7724](https://issues.apache.org/jira/browse/LUCENE-7724) by Steven Rowe (@sarowe)

Contributor guide

Open the contributing guide

Research direction

Start with TestIndexWriterExceptions.java around line 1875 and run the supplied Ant command with the recorded seed. Trace the concurrent merge and update failure through the stack frames shown, including IndexWriter and ReadersAndUpdates; done means the intermittent test failure is understood and a correction is covered by the test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.