apache / apache/lucene

AnalyzingSuggester and FuzzySuggester should be able to share same FST [LUCENE-5171]

Open
#6,235 1 comment 0 reactions 0 assignees View on GitHub
affects-version:4.3.1 affects-version:4.4 legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

In my code I use both suggesters for the same FST. I use AnalyzerSuggester#store() to create the FST and later on AnalyzingSuggester#load() and FuzzySuggester#load() to use it.
This approach works very well but it unnecessarily creates 2 fst instances resulting in 2x memory consumption.

It seems that for the time being both suggesters use the same FST format.

The following trivial method in AnalyzingSuggester provides the possibility to share the same FST among different instances of AnalyzingSuggester. It has been tested in the above scenario:

public boolean shareFstFrom(AnalyzingSuggester instance)
{
if (instance.fst == null) {
return false;
}
this.fst = instance.fst;
this.maxAnalyzedPathsForOneInput = instance.maxAnalyzedPathsForOneInput;
this.hasPayloads = instance.hasPayloads;

return true;
}

One could use it like this:

analyzingSugg = new AnalyzingSuggester(...);
fuzzySugg = new FuzzySuggester(...);
analyzingSugg.load(someInputStream);
fuzzySugg = analyzingSugg.shareFstFrom(analyzingSugg);

---
Migrated from [LUCENE-5171](https://issues.apache.org/jira/browse/LUCENE-5171) by Anna Björk Nikulásdóttir, updated Aug 16 2013

Contributor guide

Open the contributing guide

Research direction

Start by reading AnalyzingSuggester#load(), FuzzySuggester#load(), and the proposed shareFstFrom entry point, then compare the FST-related state each suggester uses. Done means both suggesters can use one loaded FST without duplicate memory while retaining their required configuration and behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.