apache / apache/lucene

Better wire up HNSW concurrent merge config

Open
#12,740 3 comments 0 reactions 0 assignees View on GitHub
type:task vector-based-search
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Follow up of #12660, currently the HNSW concurrent merge need 2 parameters to make it work: `numMergeWorker` (num of thread spawn per merge) and an `ExecutorService` that is used to execute the tasks. Currently I put them as part of `Lucene99HnswVectorFormat` such that user have to override the Codec and create their own format to use it.
We're thinking whether this could eventually be put into IWC, such that user don't need to know about Codec stuff. One thing I'm not sure is that since Lucene does not need to always use HNSW and we could very much have some other vector impl in the future. So it doesn't make sense to me if we make this config too general.
Would like to open this issue to gather some ideas, any comment is welcomed!

Contributor guide

Open the contributing guide

Research direction

Start by reviewing issue #12660 and the discussion here about numMergeWorker, ExecutorService, Lucene99HnswVectorFormat, and IWC. The issue currently asks for design feedback rather than naming a specific implementation target; done would require an agreed configuration design and defined scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.