apache / apache/lucene

CheckIndex should be concurrent [LUCENE-9662]

Open
#10,701 15 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I am watching a nightly benchmark run slowly run its `CheckIndex` step, using a single core out of the 128 cores the box has.

It seems like this is an embarrassingly parallel problem, if the index has multiple segments, and would finish much more quickly on concurrent hardware if we did "thread per segment".

If wanted to get even further concurrency, each part of the Lucene index that is checked is also independent, so it could be "thread per segment per part".

---
Migrated from [LUCENE-9662](https://issues.apache.org/jira/browse/LUCENE-9662) by Michael McCandless (@mikemccand), updated Feb 02 2022

Contributor guide

Open the contributing guide

Research direction

Start at the CheckIndex entry point and trace how checks are performed for multiple index segments and independent index parts. Review the existing benchmark scenario described in the issue, then define completion around concurrent execution that preserves CheckIndex correctness while improving its use of available cores.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.