HNSW connect components can take an inordinate amount of time
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Connect components on Flush or merge, while good for graphs that are "almost OK" but need to be better connected, can just destroy performance if the vector distribution is poor.
I don't readily have test data, but if you have tightly clustered, or many duplicate vectors, it can take until the "heat death of the universe" to complete.
It seems to me that since connect Components is a "best effort fix up" of the graph, we should add a "cap" on the amount of work this does.
Marking as a bug as I have seen this for real users and it effectively takes a CPU hostage for hours (maybe days).
### Version and environment details
_No response_
Contributor guide
Research direction
Start by locating the HNSW connect-components path invoked during flush or merge, then inspect how it behaves with tightly clustered or duplicate vectors. Reproduce the excessive runtime if possible and determine a bounded best-effort workload, with regression coverage showing that graph connection completes within the intended limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100