apache / apache/lucene

Use Suffix Arrays for fast search with leading asterisks [LUCENE-7639]

Open
#8,690 15 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

If query term starts with asterisks FST checks all words in the dictionary so request processing speed falls down. This problem can be solved with Suffix Array approach. Luckily, Suffix Array can be constructed after Lucene start from existing index. Unfortunately, Suffix Arrays requires a lot of RAM so we can use it only when special flag is set:

-Dsolr.suffixArray.enable=true

It is possible to speed up Suffix Array initialization using several threads, so we can control number of threads with

-Dsolr.suffixArray.initialization_treads_count=5

This system property can be omitted, the default value is 5.

Attached patch is the suggested implementation for SuffixArray support, it works for all terms starting with asterisks with at least 3 consequent non-wildcard characters. This patch do not change search results and affects only performance issues.

**Update**
suffix-arra-2.patch is an improved version of the first patch, system properties for it are following::

`lucene.suffixArray.enable` - `true`, if you want to enable Suffix Array support. Default value - `false`.
`lucene.suffixArray.initializationThreadsCount` - number of threads for Suffix Array initialization, if you set `0` - no additional threads used. Default value - `5`.

---
Migrated from [LUCENE-7639](https://issues.apache.org/jira/browse/LUCENE-7639) by Yakov Sirotkin, 1 vote, updated Jun 03 2017
Attachments: [suffix-array.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7639/suffix-array.patch), [suffix-array-2.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7639/suffix-array-2.patch)
Linked issues:
- #8914

Contributor guide

Open the contributing guide

Research direction

Start by reading the attached suffix-array-2.patch and the linked #8914 issue, then trace the existing FST handling for terms beginning with asterisks. Done means suffix-array support is available behind the documented lucene.suffixArray.enable and lucene.suffixArray.initializationThreadsCount properties without changing search results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.