AnalyzingInfixSuggester needs duplicate handling [LUCENE-6336]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Spinoff from #6895 but else unrelated.
Using `AnalyzingInfixSuggester` which is backed by a Lucene index and stores payload and score together with the suggest text.
I did some testing with Solr, producing the DocumentDictionary from an index with multiple documents containing the same text, but with random weights between 0-100. Then I got duplicate identical suggestions sorted by weight:
```Java
{
"suggest":{"languages":{
"engl":{
"numFound":101,
"suggestions":[{
"term":"English",
"weight":100,
"payload":"0"},
{
"term":"English",
"weight":99,
"payload":"0"},
{
"term":"English",
"weight":98,
"payload":"0"},
---etc all the way down to 0---
```
I also reproduced the same behavior in AnalyzingInfixSuggester directly. So there is a need for some duplicate removal here, either while building the local suggest index or during lookup. Only the highest weight suggestion for a given term should be returned.
---
Migrated from [LUCENE-6336](https://issues.apache.org/jira/browse/LUCENE-6336) by Jan Høydahl (@janhoy), 12 votes, updated Aug 26 2020
Attachments: [LUCENE-6336.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-6336/LUCENE-6336.patch)
Contributor guide
Research direction
Start by locating AnalyzingInfixSuggester and reproduce the duplicate-suggestion behavior described in the issue, including the direct suggester case. Compare duplicate removal during local suggest-index building with removal during lookup; done means identical terms return only once, retaining the highest weight. The issue also links LUCENE-6336.patch for prior context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100