apache / apache/lucene

Java implementation (and improvement) of Levenshtein & associated lexicon automata [LUCENE-4947]

Open
#6,011 28 comments 0 reactions 0 assignees View on GitHub
affects-version:4.0 affects-version:4.0-ALPHA affects-version:4.0-BETA affects-version:4.1 affects-version:4.2 affects-version:4.2.1 legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I was encouraged by Mike McCandless to open an issue concerning this after I contacted him privately about it. Thanks Mike!

I'd like to submit my Java implementation of the Levenshtein Automaton as a homogenous replacement for the current heterogenous, multi-component implementation in Lucene.

Benefits of upgrading include

- Reduced code complexity
- Better performance from components that were previously implemented in Python
- Support for on-the-fly dictionary-automaton manipulation (if you wish to use my dictionary-automaton implementation)

The code for all the components is well structured, easy to follow, and extensively commented. It has also been fully tested for correct functionality and performance.

The levenshtein automaton implementation (along with the required MDAG reference) can be found in my LevenshteinAutomaton Java library here: https://github.com/klawson88/LevenshteinAutomaton.

The minimalistic directed acyclic graph (MDAG) which the automaton code uses to store and step through word sets can be found here: https://github.com/klawson88/MDAG

\*-Transpositions aren't currently implemented. I hope the comment filled, editing-friendly code combined with the fact that the section in the Mihov paper detailing transpositions is only 2 pages makes adding the functionality trivial.- Update introduces transposition inclusion in edit distance calculations!

\*As a result of support for on-the-fly manipulation, the MDAG (dictionary-automaton) creation process incurs a slight speed penalty. In order to have the best of both worlds, i'd recommend the addition of a constructor which only takes sorted input. The complete, easy to follow pseudo-code for the simple procedure can be found in the first article I linked under the references section in the MDAG repository)

---
Migrated from [LUCENE-4947](https://issues.apache.org/jira/browse/LUCENE-4947) by Kevin Lawson, updated May 03 2013
Attachments: [LevenshteinAutomaton-master.zip](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4947/LevenshteinAutomaton-master.zip), [LevenshteinAutomaton-master-update.zip](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4947/LevenshteinAutomaton-master-update.zip), [MDAG-master.zip](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-4947/MDAG-master.zip)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the attached LevenshteinAutomaton and MDAG archives and the linked LevenshteinAutomaton and MDAG repositories. Compare their Java components with Lucene's current multi-component implementation, then identify the Lucene entry points and tests that would need coverage. Done means a maintained Java replacement is integrated, including the requested automaton and dictionary behavior, with functionality and performance tests passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.