apache / apache/lucene

Incremental doc values increase open file handle count significantly

Open
#16,644 2 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

The changes made in #16418 affect the file handle count significantly. Our handle-limiting guard caught this here:

https://ci-builds.apache.org/job/Lucene/job/Lucene-Check-main/16971/

Local repro on main:
```
gradlew :lucene:core:test --tests "org.apache.lucene.index.TestMixedDocValuesUpdates.testStressMultiThreading" -Ptests.asserts=false -Ptests.file.encoding=UTF-8 -Ptests.gui=true -Ptests.haltonfailure=false -Ptests.jvmargs= -Ptests.jvms=4 -Ptests.multiplier=2 -Ptests.seed=E3F5824CCA864C24 -Ptests.vectorsize=512
```

Claude's analysis:

> The open-handle count of a reader now scales as segments times updated fields times (1 + overlays) times 2. With the default of 16 overlays, an index with 50 segments and 10 hot doc-values fields could hold on the order of 17,000 descriptors instead of 1,000. I think this is worth raising with Jim on the PR or a new issue rather than just bumping the test limit.

I wonder if this is something that should be fixed in the patch/core rather than just by bumping this test's `HandleLimitFS.MaxOpenHandles`. I bet this will affect people if it's true.

@jimczi ?

### Version and environment details

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by running the provided Gradle command for org.apache.lucene.index.TestMixedDocValuesUpdates.testStressMultiThreading and inspect the changes from #16418. Trace the file handles opened during incremental doc-values updates and review HandleLimitFS.MaxOpenHandles in that test. Done means the stress test no longer shows the reported handle-count increase without simply raising its limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.