apache / apache/lucene

A new token filter: SubSequence [LUCENE-5674]

Open
#6,736 18 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor module:core/other type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

A new configurable token filter which, given a token breaks it into sub-parts and outputs consecutive sub-sequences of those sub-parts.

Useful for, for example, using during indexing to generate variations on domain names, so that "www.google.com" can be found by searching for "google.com", or "www.google.com".

Parameters:

sepRegexp: A regular expression used split incoming tokens into sub-parts.
glue: A string used to concatenate sub-parts together when creating sub-sequences.
minLen: Minimum length (in sub-parts) of output sub-sequences
maxLen: Maximum length (in sub-parts) of output sub-sequences (0 for unlimited; negative numbers for token length in sub-parts minus specified length)
anchor: Anchor.START to output only prefixes, or Anchor.END to output only suffixes, or Anchor.NONE to output any sub-sequence
withOriginal: whether to output also the original token

EDIT: now includes tests for filter and for factory.

---
Migrated from [LUCENE-5674](https://issues.apache.org/jira/browse/LUCENE-5674) by Nitzan Shaked, updated Jun 01 2014
Attachments: [subseqfilter.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5674/subseqfilter.patch)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the attached subseqfilter.patch and the issue's stated filter parameters, then locate the existing token-filter and factory tests referenced in the issue. Verify the proposed behavior for prefixes, suffixes, arbitrary sub-sequences, length limits, separators, and original-token output; done means the filter and factory tests cover and pass those cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.