A new token filter: SubSequence [LUCENE-5674]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
A new configurable token filter which, given a token breaks it into sub-parts and outputs consecutive sub-sequences of those sub-parts.
Useful for, for example, using during indexing to generate variations on domain names, so that "www.google.com" can be found by searching for "google.com", or "www.google.com".
Parameters:
sepRegexp: A regular expression used split incoming tokens into sub-parts.
glue: A string used to concatenate sub-parts together when creating sub-sequences.
minLen: Minimum length (in sub-parts) of output sub-sequences
maxLen: Maximum length (in sub-parts) of output sub-sequences (0 for unlimited; negative numbers for token length in sub-parts minus specified length)
anchor: Anchor.START to output only prefixes, or Anchor.END to output only suffixes, or Anchor.NONE to output any sub-sequence
withOriginal: whether to output also the original token
EDIT: now includes tests for filter and for factory.
---
Migrated from [LUCENE-5674](https://issues.apache.org/jira/browse/LUCENE-5674) by Nitzan Shaked, updated Jun 01 2014
Attachments: [subseqfilter.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5674/subseqfilter.patch)
Contributor guide
Research direction
Start by reviewing the attached subseqfilter.patch and the issue's stated filter parameters, then locate the existing token-filter and factory tests referenced in the issue. Verify the proposed behavior for prefixes, suffixes, arbitrary sub-sequences, length limits, separators, and original-token output; done means the filter and factory tests cover and pass those cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100