apache / apache/lucene

HNSW connect components can take an inordinate amount of time

Open
#14,214 19 comments 3 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Connect components on Flush or merge, while good for graphs that are "almost OK" but need to be better connected, can just destroy performance if the vector distribution is poor.

I don't readily have test data, but if you have tightly clustered, or many duplicate vectors, it can take until the "heat death of the universe" to complete.

It seems to me that since connect Components is a "best effort fix up" of the graph, we should add a "cap" on the amount of work this does.

Marking as a bug as I have seen this for real users and it effectively takes a CPU hostage for hours (maybe days).

### Version and environment details

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the HNSW connect-components path invoked during flush or merge, then inspect how it behaves with tightly clustered or duplicate vectors. Reproduce the excessive runtime if possible and determine a bounded best-effort workload, with regression coverage showing that graph connection completes within the intended limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.