apache / apache/lucene

Speed up Levenshtein distance calculation when we don't need the exact distance [LUCENE-9289]

Open
#10,329 3 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor module:suggest type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Sometimes when we calculate the Levenshtein distance we don't need the exact distance, we only want to know if the strings are similar enough.

```
sug.score = sd.getDistance(original, sug.string);
if (sug.score < min) continue;
```
If we use this threshold in the distance calculation, we can speed it up, we can stop the calculation when we already know that the the the distance will be lower than the threshold.

---
Migrated from [LUCENE-9289](https://issues.apache.org/jira/browse/LUCENE-9289) by Andras Salamon (@asalamon74), updated Mar 30 2020
Attachments: [SOLR-14360-01.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-9289/SOLR-14360-01.patch)

Contributor guide

Open the contributing guide

Research direction

Start at solr/core/src/java/org/apache/solr/spelling/SolrSpellChecker.java around lines 113-114, where getDistance is called and compared with min. Trace the distance implementation and determine how that threshold can be used during calculation. Done means the similarity check can stop when the result is already below the threshold while preserving the existing score behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.