apache / apache/lucene

A new "Diversification" Type collector akin to FirstPassGroupingCollector

Open
#15,190 0 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

FirstPassGroupingCollector is pretty awesome, but we are fairly restricted on the things that we are actually grouping by.

Search is evolving and the desire to diversify results (to feed to an LLM, or even just to show users), is getting more and more important.

I don't have a fully concrete idea, but it seems to me that Lucene should be able to support a "grouping" by some statistics or requirements of another field.

Two examples that come to mind:

- Maximum marginal relevance
- Diversification based on clusters of vectors (e.g. kmeans)

Both of these will be complicated in their own ways because the groupings end up being dynamic as more data is seen (instead of having a natural static upper limit based on cardinality).

But it seems generally useful for search.

Contributor guide

Open the contributing guide

Research direction

Start by reading FirstPassGroupingCollector and the issue's examples of maximum marginal relevance and vector-cluster diversification. The proposal does not name files, tests, or a concrete API; work is only complete once the dynamic grouping requirements and a specific Lucene design are defined and accepted.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.