apache / apache/lucene

Multi-HNSW graphs per segment?

Open
#14,341 8 comments 0 reactions 1 assignee Claimed by @atris View on GitHub
type:enhancement vector-based-search
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

What do we think about clustering or grouping documents by centroids, or potentially in chunks of filters and allow multiple graphs per segment. If segments are random sub-samples, it is conceivable that clusters are of common distributions, consequently we can quit searching clusters early and only search a couple of the clusters at a time.

Seeing the optimistic query changes, makes me think we can maybe cluster, have one graph per cluster (or chunk of the ordered docs) and more eagerly enable early termination of the multi-graph searches.

This is a crazy idea, maybe it ain't worth doing.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.