apache / apache/lucene

Add support for keeping the graph structure for HNSW in off heap during merges

Open
#15,904 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description
Based on my reading of the code in Lucene, what I can see is we keep the graph structure OnHeap which not only puts additional pressure on JVM but also creates a bottleneck where we cannot create larger graphs.

One solution I was thinking is to move the graph structure during merge time to offheap, and also during write write it directly through off heap.

I am not sure if we have tried this in past, but I was going to put a POC around this. If anyone has some thoughts or ideas please share.

Contributor guide

Open the contributing guide

Research direction

Start by tracing Lucene's HNSW graph handling during merges and writes, focusing on where the graph is kept on heap. Review any prior attempts or related implementation paths before defining the proof of concept. Done means demonstrating whether the graph can be maintained off heap without blocking larger graph creation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.