TestIndexWriterWithThreads can sometimes use too many open files
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
This is distinct from #14483, didn't happen in a nightly build but instead an s390 one.
NOTE: reproduce with: gradlew test --tests TestIndexWriterWithThreads.testCloseWithThreads -Dtests.seed=EA05EB86B7F987A6 -Dtests.multiplier=2 -Dtests.directory=MMapDirectory -Dtests.locale=jgo-CM -Dtests.timezone=Asia/Tokyo -Dtests.asserts=true -Dtests.file.encoding=UTF-8
Maybe it won't reproduce, but we can look at the test and see if it might use too many resources, especially with a multiplier. Could be we just need to apply a helper method to use less files.
```
FAILED: org.apache.lucene.index.TestIndexWriterWithThreads.testCloseWithThreads
Error Message:
java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit
Stack Trace:
java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit
at __randomizedtesting.SeedInfo.seed([EA05EB86B7F987A6:F7C1C2F1D40C9D00]:0)
at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:5603)
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3805)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:4146)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4108)
at org.apache.lucene.index.TestIndexWriterWithThreads.testCloseWithThreads(TestIndexWriterWithThreads.java:240)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
at java.base/java.lang.Thread.run(Thread.java:1575)
Caused by: java.nio.file.FileSystemException: /home/jenkins/jenkins-home/712657a4/workspace/Lucene/Lucene-MMAPv2-Linux (s390x big endian)/lucene/core/build/tmp/tests-tmp/lucene.index.TestIndexWriterWithThreads_EA05EB86B7F987A6-001/index-MMapDirectory-014/_7a_Lucene90FieldsIndex-doc_ids_t2.tmp: Too many open files
at org.apache.lucene.tests.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:67)
at org.apache.lucene.tests.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
at org.apache.lucene.tests.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:163)
at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:200)
at java.base/java.nio.file.Files.newOutputStream(Files.java:228)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:389)
at org.apache.lucene.store.FSDirectory.createTempOutput(FSDirectory.java:229)
at org.apache.lucene.tests.store.MockDirectoryWrapper.createTempOutput(MockDirectoryWrapper.java:753)
at org.apache.lucene.store.FilterDirectory.createTempOutput(FilterDirectory.java:81)
at org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:49)
at org.apache.lucene.codecs.lucene90.compressing.FieldsIndexWriter.(FieldsIndexWriter.java:83)
at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.(Lucene90CompressingStoredFieldsWriter.java:152)
at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsFormat.fieldsWriter(Lucene90CompressingStoredFieldsFormat.java:140)
at org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat.fieldsWriter(Lucene90StoredFieldsFormat.java:154)
at org.apache.lucene.tests.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
at org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:50)
at org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:57)
at org.apache.lucene.index.IndexingChain.startStoredFields(IndexingChain.java:544)
at org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:575)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:274)
at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:425)
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1572)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1857)
at org.apache.lucene.index.TestIndexWriterWithThreads$IndexerThread.run(TestIndexWriterWithThreads.java:93)
```
### Version and environment details
_No response_
Contributor guide
Research direction
Start with TestIndexWriterWithThreads.java, especially testCloseWithThreads and IndexerThread.run around the reported lines, then run the supplied Gradle command and seed. Inspect how the test and MMapDirectory use open files under the multiplier; done means the test no longer hits the HandleLimitFS “Too many open files” failure in the reported configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100