apache / apache/lucene

TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnCommit fails on nightly

Open
#14,261 0 comments 0 reactions 0 assignees View on GitHub
type:test
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

This fails with an OOM on nightly. I wonder if we are just merging too much :).

I suspect this is due to the tiered merge policy change, but I might be wrong. haven't ran gitbisect.

```
TestIndexWriterMergePolicy > testStressUpdateSameDocumentWithMergeOnCommit FAILED
java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.apache.lucene.util.ArrayUtil.growExact(ArrayUtil.java:320)
at org.apache.lucene.util.ArrayUtil.growInRange(ArrayUtil.java:346)
at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:354)
at org.apache.lucene.util.BytesRefArray.append(BytesRefArray.java:67)
at org.apache.lucene.index.FieldUpdatesBuffer.append(FieldUpdatesBuffer.java:204)
at org.apache.lucene.index.FieldUpdatesBuffer.addUpdate(FieldUpdatesBuffer.java:173)
at org.apache.lucene.index.BufferedUpdates.addNumericUpdate(BufferedUpdates.java:139)
at org.apache.lucene.index.DocumentsWriterDeleteQueue$DocValuesUpdatesNode.apply(DocumentsWriterDeleteQueue.java:504)
at org.apache.lucene.index.DocumentsWriterDeleteQueue$DeleteSlice.apply(DocumentsWriterDeleteQueue.java:364)
at org.apache.lucene.index.DocumentsWriterPerThread.prepareFlush(DocumentsWriterPerThread.java:398)
at org.apache.lucene.index.DocumentsWriter.lambda$doFlush$9(DocumentsWriter.java:497)
at org.apache.lucene.index.DocumentsWriter$$Lambda/0x000003fc011b3758.get(Unknown Source)
at org.apache.lucene.index.DocumentsWriterFlushQueue.addTicket(DocumentsWriterFlushQueue.java:43)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:496)
at org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:649)
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3677)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:4125)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4087)
at org.apache.lucene.tests.index.RandomIndexWriter.commit(RandomIndexWriter.java:414)
at org.apache.lucene.tests.index.RandomIndexWriter.commit(RandomIndexWriter.java:370)
at org.apache.lucene.index.TestIndexWriterMergePolicy.stressUpdateSameDocumentWithMergeOnX(TestIndexWriterMergePolicy.java:808)
at org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnCommit(TestIndexWriterMergePolicy.java:747)
at java.base/java.lang.invoke.LambdaForm$DMH/0x000003fc01128000.invokeVirtual(LambdaForm$DMH)
at java.base/java.lang.invoke.LambdaForm$MH/0x000003fc010a8800.invoke(LambdaForm$MH)
at java.base/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:153)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=20, name=Thread-3, state=RUNNABLE, group=TGRP-TestIndexWriterMergePolicy]

Caused by:
java.lang.OutOfMemoryError: GC overhead limit exceeded
at __randomizedtesting.SeedInfo.seed([C13C284249671712]:0)
at org.apache.lucene.util.ArrayUtil.growExact(ArrayUtil.java:320)
at org.apache.lucene.util.ArrayUtil.growInRange(ArrayUtil.java:346)
at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:354)
at org.apache.lucene.index.FieldUpdatesBuffer.add(FieldUpdatesBuffer.java:144)
at org.apache.lucene.index.FieldUpdatesBuffer.addUpdate(FieldUpdatesBuffer.java:175)
at org.apache.lucene.index.BufferedUpdates.addNumericUpdate(BufferedUpdates.java:139)
at org.apache.lucene.index.DocumentsWriterDeleteQueue$DocValuesUpdatesNode.apply(DocumentsWriterDeleteQueue.java:504)
at org.apache.lucene.index.DocumentsWriterDeleteQueue$DeleteSlice.apply(DocumentsWriterDeleteQueue.java:364)
at org.apache.lucene.index.DocumentsWriterPerThread.prepareFlush(DocumentsWriterPerThread.java:398)
at org.apache.lucene.index.DocumentsWriter.lambda$doFlush$9(DocumentsWriter.java:497)
at org.apache.lucene.index.DocumentsWriter$$Lambda/0x000003fc011b3758.get(Unknown Source)
at org.apache.lucene.index.DocumentsWriterFlushQueue.addTicket(DocumentsWriterFlushQueue.java:43)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:496)
at org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456)
at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:404)
at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:447)
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1552)
at org.apache.lucene.index.IndexWriter.softUpdateDocument(IndexWriter.java:1870)
at org.apache.lucene.tests.index.RandomIndexWriter.updateDocument(RandomIndexWriter.java:319)
at org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:781)
at org.apache.lucene.index.TestIndexWriterMergePolicy$$Lambda/0x000003fc011e5438.run(Unknown Source)
at java.base/java.lang.Thread.runWith(Thread.java:1583)
at java.base/java.lang.Thread.run(Thread.java:1570)

java.lang.AssertionError: org.apache.lucene.store.AlreadyClosedException: This DocumentsWriterDeleteQueue is already closed
at __randomizedtesting.SeedInfo.seed([C13C284249671712]:0)
at org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:787)
at java.base/java.lang.Thread.run(Thread.java:1570)

Caused by:
org.apache.lucene.store.AlreadyClosedException: This DocumentsWriterDeleteQueue is already closed
at app//org.apache.lucene.index.DocumentsWriterDeleteQueue.ensureOpen(DocumentsWriterDeleteQueue.java:308)
at app//org.apache.lucene.index.DocumentsWriterDeleteQueue.freezeGlobalBuffer(DocumentsWriterDeleteQueue.java:220)
at app//org.apache.lucene.index.DocumentsWriterPerThread.prepareFlush(DocumentsWriterPerThread.java:393)
at app//org.apache.lucene.index.DocumentsWriter.lambda$doFlush$9(DocumentsWriter.java:497)
at app//org.apache.lucene.index.DocumentsWriterFlushQueue.addTicket(DocumentsWriterFlushQueue.java:43)
at app//org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:496)
at app//org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:456)
at app//org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:391)
at app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:413)
at app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1552)
at app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1837)
at app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1477)
at app//org.apache.lucene.tests.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:224)
at app//org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:783)
... 1 more

java.lang.AssertionError: org.apache.lucene.store.AlreadyClosedException: flush control is closed
at __randomizedtesting.SeedInfo.seed([C13C284249671712]:0)
at org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:787)
at java.base/java.lang.Thread.run(Thread.java:1570)

Caused by:
org.apache.lucene.store.AlreadyClosedException: flush control is closed
at app//org.apache.lucene.index.DocumentsWriterFlushControl.obtainAndLock(DocumentsWriterFlushControl.java:552)
at app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:415)
at app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1552)
at app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1837)
at app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1477)
at app//org.apache.lucene.tests.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:224)
at app//org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:783)
... 1 more

java.lang.AssertionError: pendingNumDocs 60573 != 60476 totalMaxDoc
at __randomizedtesting.SeedInfo.seed([C13C284249671712]:0)
at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2463)
at org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:5765)
at org.apache.lucene.index.IndexWriter.tragicEvent(IndexWriter.java:5755)
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1556)
at org.apache.lucene.index.IndexWriter.softUpdateDocuments(IndexWriter.java:1597)
at org.apache.lucene.tests.index.RandomIndexWriter.updateDocument(RandomIndexWriter.java:315)
at org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$4(TestIndexWriterMergePolicy.java:781)
at java.base/java.lang.Thread.run(Thread.java:1570)
```

### Gradle command to reproduce

```
./gradlew test --tests TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnCommit -Dtests.seed=C13C284249671712 -Dtests.nightly=true -Dtests.locale=rwk-TZ -Dtests.timezone=Antarctica/Macquarie -Dtests.asserts=true -Dtests.file.encoding=UTF-8
```

Contributor guide

Open the contributing guide

Research direction

Run the supplied Gradle command with the listed seed and nightly settings, then read TestIndexWriterMergePolicy.java around testStressUpdateSameDocumentWithMergeOnCommit and stressUpdateSameDocumentWithMergeOnX. Trace the reported merge, flush, and FieldUpdatesBuffer frames to identify the cause; done means the nightly test completes without the OOM or follow-on failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.