apache / apache/lucene

TestIndexWriterMergePolicy opens too many file handles at night

Open
#14,483 3 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Haven't dug in yet: this test probably just needs tweaking to use less files, there are some helper methods we can use to do it.

```
Build: https://jenkins.thetaphi.de/job/Lucene-nightly-main-Linux/9/
Java: 64bit/hotspot/jdk-24 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED: org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnCommit

Error Message:
java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit

Stack Trace:
java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot commit
at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:5603)
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3805)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:4146)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4108)
at org.apache.lucene.tests.index.RandomIndexWriter.commit(RandomIndexWriter.java:388)
at org.apache.lucene.tests.index.RandomIndexWriter.commit(RandomIndexWriter.java:370)
at org.apache.lucene.index.TestIndexWriterMergePolicy.stressUpdateSameDocumentWithMergeOnX(TestIndexWriterMergePolicy.java:808)
at org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnCommit(TestIndexWriterMergePolicy.java:747)
...
Caused by: java.nio.file.FileSystemException: /home/jenkins/workspace/Lucene-nightly-main-Linux/lucene/core/build/tmp/tests-tmp/lucene.index.TestIndexWriterMergePolicy_D1A5025C440EEC99-001/index-MMapDirectory-001/_5yt.fdt: Too many open files
at org.apache.lucene.tests.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:67)
at org.apache.lucene.tests.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
at org.apache.lucene.tests.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:163)
at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:200)
at java.base/java.nio.file.Files.newOutputStream(Files.java:215)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:389)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:382)
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:215)
at org.apache.lucene.tests.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:718)
at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:41)
at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.(Lucene90CompressingStoredFieldsWriter.java:138)
at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsFormat.fieldsWriter(Lucene90CompressingStoredFieldsFormat.java:140)
at org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat.fieldsWriter(Lucene90StoredFieldsFormat.java:154)
at org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:50)
at org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:57)
at org.apache.lucene.index.IndexingChain.startStoredFields(IndexingChain.java:544)
at org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:575)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:274)
at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:425)
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1572)
at org.apache.lucene.index.IndexWriter.softUpdateDocument(IndexWriter.java:1890)
at org.apache.lucene.tests.index.RandomIndexWriter.updateDocument(RandomIndexWriter.java:319)
at org.apache.lucene.index.TestIndexWriterMergePolicy.lambda$stressUpdateSameDocumentWithMergeOnX$0(TestIndexWriterMergePolicy.java:781)
... 1 more
```

### Version and environment details

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by running TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnCommit and inspect stressUpdateSameDocumentWithMergeOnX in TestIndexWriterMergePolicy.java, around the reported failure. Review the available helper methods and reduce the test's file usage; done means the test passes without triggering the HandleLimitFS too-many-open-files error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search, testing
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.