apache / apache/lucene

Could/should KNN queries use per-segment query caching?

Open
#14,669 2 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

@msokolov and I were talking about how Lucene's KNN queries don't do any per-segment caching because they do all of their work up front in `rewrite`, which is before per-segment query cache is checked (I think)...

But I think conceptually, the way the optimistic KNN query rewrite works, where it asks each segment for its top N (N = pro-rated estimate from that segment's size vs whole index, and the top K requested), merge sorts all of those, and then goes back to any segment(s) that might still have further competitive hits and digs deeper, ... this should work well with caching, if we could somehow implement it?

I.e. each segment would cache Query (vector + top K) to sparse bitset, and then a future identical KNN query could pull from that cache if its requested top K/N fits within what's already cached?

But, would vector queries have cache hits? At Amazon Product Search we (strangely) can send the identical vector multiple times even within the execution of a single end-Customer query ... would other Lucene apps have similar behavior? E.g. if they are running inference on something simple, like just the search terms from their users, the identical vector could indeed appear? Or maybe this cache wouldn't have to have precisely identical vectors ... maybe vectors that are within some small distance from a cached vector should count as a hit too?

Anyway, no clear way forward here, but I wanted to at least open a discussion in case others have ideas around Lucene's per-segment query caching and KNN queries ...

Contributor guide

Open the contributing guide

Research direction

Start by examining Lucene's KNN query rewrite and per-segment query-cache paths, focusing on how optimistic rewrite gathers and merges segment results. Investigate whether a segment can safely cache a vector query's top-K results and reuse them for larger requests, then validate the trade-offs with repeated identical vectors and nearby vectors. Done requires a documented, technically supported direction rather than an unbounded discussion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.