Add support for keeping the graph structure for HNSW in off heap during merges
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Based on my reading of the code in Lucene, what I can see is we keep the graph structure OnHeap which not only puts additional pressure on JVM but also creates a bottleneck where we cannot create larger graphs.
One solution I was thinking is to move the graph structure during merge time to offheap, and also during write write it directly through off heap.
I am not sure if we have tried this in past, but I was going to put a POC around this. If anyone has some thoughts or ideas please share.
Contributor guide
Research direction
Start by tracing Lucene's HNSW graph handling during merges and writes, focusing on where the graph is kept on heap. Review any prior attempts or related implementation paths before defining the proof of concept. Done means demonstrating whether the graph can be maintained off heap without blocking larger graph creation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100