apache / apache/lucene

Spellchecker don't find suggestion for concrete misspelled 6 letter words [LUCENE-4793]

Open
#5,858 1 comment 0 reactions 0 assignees View on GitHub
affects-version:3.6 affects-version:4.0 affects-version:4.1 legacy-jira-priority:Minor module:suggest type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 12h
Merged PRs (30d)
86

Description

Debugging Solr spellchecker (IndexBasedSpellchecker, delegating on lucene Spellchecker) behaviour i think i found a bug when the input is a 6 letter word:
- george
- anthem
- argued
- fluent

Due to the getMin() and getMax() the grams indexed for these terms are 3 and 4. So, the fields would be something like this:
- for "**george**"
- start3: "geo"
- start4: "geor"
- end3: "rge"
- end4: "orge"
- 3: "geo", "eor", "org", "rge"
- 4: "geor", "eorg", "orge"
- for "**anthem**"
- start3: "ant"
- start4: "anth"
- end3: "tem"
- end4: "them"

The problem shows up when the user swap 3rd a 4th characters, misspelling the word like this:
- geroge
- anhtem

The queries generated for this terms are: (SHOULD boolean queries)
- for "**geroge**"
- start3: "ger"
- start4: "gero"
- end3: "oge"
- end4: "roge"
- 3: "ger", "ero", "rog", "oge"
- 4: "gero", "erog", "roge"
- for "**anhtem**"
- start3: "anh"
- start4: "anht"
- end3: "tem"
- end4: "htem"
- 3: "anh", "nht", "hte", "tem"
- 4: "anht", "nhte", "htem"

So, as you can see, this kind of misspelling never matches the suitable suggestions although the edit distance is 0.95555556.

I think getMin(int l) and getMax(int l) should return 2 and 3, respectively, for l==6. Debugging other values i did not found any problem with any kind of misspelling.

---
Migrated from [LUCENE-4793](https://issues.apache.org/jira/browse/LUCENE-4793) by Samuel García Martínez (@samuelgmartinez)
Attachments: [SpellcheckerTest.java](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4793/SpellcheckerTest.java)

Contributor guide

Open the contributing guide

Research direction

Start with the attached SpellcheckerTest.java and trace the Lucene spellchecker code that implements getMin(int l) and getMax(int l). Reproduce the six-letter transposition examples such as “geroge” and “anhtem”, then add or update a regression test showing that suitable suggestions are found for these inputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
40/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.