minor hunspell optimizations [LUCENE-5518]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
After benchmarking indexing speed on [SOLR-3245](https://issues.apache.org/jira/browse/SOLR-3245), I ran a profiler and a couple things stood out.
There are other things I want to improve too, but these almost double the speed for many dictionaries.
- Hunspell supports two-stage affix stripping, but the vast majority of dictionaries don't have any affixes that support it. So we just add a boolean (Dictionary.twoStageAffix) that is false until we see one.
- We use java.util.regex.Pattern for condition checks. This is slow, I switched to o.a.l.automaton and its much faster, and uses slightly less RAM too.
---
Migrated from [LUCENE-5518](https://issues.apache.org/jira/browse/LUCENE-5518) by Robert Muir (@rmuir), updated Mar 12 2014
Attachments: [LUCENE-5518.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5518/LUCENE-5518.patch) (versions: 2)
Contributor guide
Research direction
Start by reviewing the attached LUCENE-5518.patch and the Hunspell indexing and condition-checking code it changes. Verify the two-stage-affix fast path and automaton-based conditions against the existing behavior, then benchmark indexing to confirm the reported speed and memory improvements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance, search
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100