TestIndexWriterMergePolicy opens too many file handles at night
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Haven't dug in yet: this test probably just needs tweaking to use less files, there are some helper methods we can use to do it.
```
Build: https://jenkins.thetaphi.de/job/Lucene-nightly-main-Linux/9/
Java: 64bit/hotspot/jdk-24 -XX:-UseCompressedOops -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnCommit
Error Message:
java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit
Stack Trace:
java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit
at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:5603)
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3805)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:4146)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4108)
at org.apache.lucene.tests.index.RandomIndexWriter.commit(RandomIndexWriter.java:388)
at org.apache.lucene.tests.index.RandomIndexWriter.commit(RandomIndexWriter.java:370)
at org.apache.lucene.index.TestIndexWriterMergePolicy.stressUpdateSameDocumentWithMergeOnX(TestIndexWriterMergePolicy.java:808)
at org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnCommit(TestIndexWriterMergePolicy.java:747)
...
Caused by: java.nio.file.FileSystemException: /home/jenkins/workspace/Lucene-nightly-main-Linux/lucene/core/build/tmp/tests-tmp/lucene.index.TestIndexWriterMergePolicy_D1A5025C440EEC99-001/index-MMapDirectory-001/_5yt.fdt: Too many open files
at org.apache.lucene.tests.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:67)
at org.apache.lucene.tests.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
at org.apache.lucene.tests.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:163)
at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:200)
at java.base/java.nio.file.Files.newOutputStream(Files.java:215)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:389)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:382)
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:215)
at org.apache.lucene.tests.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:718)
at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:41)
at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.(Lucene90CompressingStoredFieldsWriter.java:138)
at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsFormat.fieldsWriter(Lucene90CompressingStoredFieldsFormat.java:140)
at org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat.fieldsWriter(Lucene90StoredFieldsFormat.java:154)
at org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:50)
at org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:57)
at org.apache.lucene.index.IndexingChain.startStoredFields(IndexingChain.java:544)
at org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:575)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:274)
at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:425)
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1572)
at org.apache.lucene.index.IndexWriter.softUpdateDocument(IndexWriter.java:1890)
at org.apache.lucene.tests.index.RandomIndexWriter.updateDocument(RandomIndexWriter.java:319)
at org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$0(TestIndexWriterMergePolicy.java:781)
... 1 more
```
### Version and environment details
_No response_
Contributor guide
Research direction
Start by running TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnCommit and inspect stressUpdateSameDocumentWithMergeOnX in TestIndexWriterMergePolicy.java, around the reported failure. Review the available helper methods and reduce the test's file usage; done means the test passes without triggering the HandleLimitFS too-many-open-files error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search, testing
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100