Make preserving original token in CompoundWordTokenFilterBase configurable [LUCENE-9585]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
When using a subclass of CompoundWordTokenFilterBase the filter will always output the original input token along with the decomposed tokens if there are any. This will result in documents that originally had the compound form to have both the compound and decomposed form while documents that originally had the decomposed form will only have the decomposed form. Only queries in the decomposed forms will match more documents when using this filter.
If the filter can also be run at query time compound forms can be decomposed and match additional documents. To do this the filter needs to be able to return only the decomposed form if there is a decomposed form.
---
Migrated from [LUCENE-9585](https://issues.apache.org/jira/browse/LUCENE-9585) by Geoffrey Lawson (@glawson0), updated Oct 30 2020
Contributor guide
Research direction
Start by locating CompoundWordTokenFilterBase and reviewing how it currently emits the original and decomposed tokens. Define the configurable behavior for retaining or omitting the original token, then add coverage showing both modes, including query-time decomposition behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100