apache / apache/lucene

minor hunspell optimizations [LUCENE-5518]

Open
#6,581 4 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

After benchmarking indexing speed on [SOLR-3245](https://issues.apache.org/jira/browse/SOLR-3245), I ran a profiler and a couple things stood out.

There are other things I want to improve too, but these almost double the speed for many dictionaries.

- Hunspell supports two-stage affix stripping, but the vast majority of dictionaries don't have any affixes that support it. So we just add a boolean (Dictionary.twoStageAffix) that is false until we see one.
- We use java.util.regex.Pattern for condition checks. This is slow, I switched to o.a.l.automaton and its much faster, and uses slightly less RAM too.

---
Migrated from [LUCENE-5518](https://issues.apache.org/jira/browse/LUCENE-5518) by Robert Muir (@rmuir), updated Mar 12 2014
Attachments: [LUCENE-5518.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5518/LUCENE-5518.patch) (versions: 2)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the attached LUCENE-5518.patch and the Hunspell indexing and condition-checking code it changes. Verify the two-stage-affix fast path and automaton-based conditions against the existing behavior, then benchmark indexing to confirm the reported speed and memory improvements.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance, search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.