apache / apache/lucene

Make preserving original token in CompoundWordTokenFilterBase configurable [LUCENE-9585]

Open
#10,625 1 comment 0 reactions 0 assignees View on GitHub
affects-version:8.5.1 legacy-jira-priority:Minor module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

When using a subclass of CompoundWordTokenFilterBase the filter will always output the original input token along with the decomposed tokens if there are any. This will result in documents that originally had the compound form to have both the compound and decomposed form while documents that originally had the decomposed form will only have the decomposed form. Only queries in the decomposed forms will match more documents when using this filter.

If the filter can also be run at query time compound forms can be decomposed and match additional documents. To do this the filter needs to be able to return only the decomposed form if there is a decomposed form.

---
Migrated from [LUCENE-9585](https://issues.apache.org/jira/browse/LUCENE-9585) by Geoffrey Lawson (@glawson0), updated Oct 30 2020

Contributor guide

Open the contributing guide

Research direction

Start by locating CompoundWordTokenFilterBase and reviewing how it currently emits the original and decomposed tokens. Define the configurable behavior for retaining or omitting the original token, then add coverage showing both modes, including query-time decomposition behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.