Better wire up HNSW concurrent merge config
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Follow up of #12660, currently the HNSW concurrent merge need 2 parameters to make it work: `numMergeWorker` (num of thread spawn per merge) and an `ExecutorService` that is used to execute the tasks. Currently I put them as part of `Lucene99HnswVectorFormat` such that user have to override the Codec and create their own format to use it.
We're thinking whether this could eventually be put into IWC, such that user don't need to know about Codec stuff. One thing I'm not sure is that since Lucene does not need to always use HNSW and we could very much have some other vector impl in the future. So it doesn't make sense to me if we make this config too general.
Would like to open this issue to gather some ideas, any comment is welcomed!
Contributor guide
Research direction
Start by reviewing issue #12660 and the discussion here about numMergeWorker, ExecutorService, Lucene99HnswVectorFormat, and IWC. The issue currently asks for design feedback rather than naming a specific implementation target; done would require an agreed configuration design and defined scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100