Incremental doc values increase open file handle count significantly
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
The changes made in #16418 affect the file handle count significantly. Our handle-limiting guard caught this here:
https://ci-builds.apache.org/job/Lucene/job/Lucene-Check-main/16971/
Local repro on main:
```
gradlew :lucene:core:test --tests "org.apache.lucene.index.TestMixedDocValuesUpdates.testStressMultiThreading" -Ptests.asserts=false -Ptests.file.encoding=UTF-8 -Ptests.gui=true -Ptests.haltonfailure=false -Ptests.jvmargs= -Ptests.jvms=4 -Ptests.multiplier=2 -Ptests.seed=E3F5824CCA864C24 -Ptests.vectorsize=512
```
Claude's analysis:
> The open-handle count of a reader now scales as segments times updated fields times (1 + overlays) times 2. With the default of 16 overlays, an index with 50 segments and 10 hot doc-values fields could hold on the order of 17,000 descriptors instead of 1,000. I think this is worth raising with Jim on the PR or a new issue rather than just bumping the test limit.
I wonder if this is something that should be fixed in the patch/core rather than just by bumping this test's `HandleLimitFS.MaxOpenHandles`. I bet this will affect people if it's true.
@jimczi ?
### Version and environment details
_No response_
Contributor guide
Research direction
Start by running the provided Gradle command for org.apache.lucene.index.TestMixedDocValuesUpdates.testStressMultiThreading and inspect the changes from #16418. Trace the file handles opened during incremental doc-values updates and review HandleLimitFS.MaxOpenHandles in that test. Done means the stress test no longer shows the reported handle-count increase without simply raising its limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100