CheckIndex should be concurrent [LUCENE-9662]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
I am watching a nightly benchmark run slowly run its `CheckIndex` step, using a single core out of the 128 cores the box has.
It seems like this is an embarrassingly parallel problem, if the index has multiple segments, and would finish much more quickly on concurrent hardware if we did "thread per segment".
If wanted to get even further concurrency, each part of the Lucene index that is checked is also independent, so it could be "thread per segment per part".
---
Migrated from [LUCENE-9662](https://issues.apache.org/jira/browse/LUCENE-9662) by Michael McCandless (@mikemccand), updated Feb 02 2022
Contributor guide
Research direction
Start at the CheckIndex entry point and trace how checks are performed for multiple index segments and independent index parts. Review the existing benchmark scenario described in the issue, then define completion around concurrent execution that preserves CheckIndex correctness while improving its use of available cores.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100