apache / apache/lucene

Refactor HNSW graph build such that concurrent build won't impact single thread build

Open
#12,732 2 comments 0 reactions 0 assignees View on GitHub
type:task vector-based-search
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

follow-up from #12660, in that PR I made it default that we're going to use Lock and Atomics in both single thread and multithread build. As @benwtrent pointed out it might be better to have separate `NeighbourArray/HnswGraphBuilder` implementation such that we'll separate the single thread from multithread execution and don't add the overhead to single thread build.

Contributor guide

Open the contributing guide

Research direction

Start by reading the referenced PR #12660 and the HNSW graph-building code around NeighbourArray and HnswGraphBuilder. Trace how single-threaded and concurrent builds currently share Lock and Atomics, then define separate implementations whose behavior preserves both build paths while removing unnecessary single-thread overhead.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.