apache / apache/lucene

HyphenationCompoundWordTokenFilter returns terms shorter than minSubwordSize [LUCENE-8185]

Open
#9,233 0 comments 0 reactions 0 assignees View on GitHub
affects-version:6.6.1 affects-version:7.2.1 legacy-jira-priority:Minor type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

To account for languages which use binding characters ("fogemorphemes") for composing words the `HyphenationCompoundWordTokenFilter` re-checks the dictionary for a candidate with the last character removed when the original candidate was not found. It currently does not re-check against `minSubWordSize` in this case. Terms that are one character shorter than `minSubWordSize` can be returned.

---
Migrated from [LUCENE-8185](https://issues.apache.org/jira/browse/LUCENE-8185) by Matthias Krueger (@mkr), updated Mar 06 2018
Attachments: [LUCENE-8185.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8185/LUCENE-8185.patch)
Pull requests: https://github.com/apache/lucene-solr/pull/326

Contributor guide

Open the contributing guide

Research direction

Start with HyphenationCompoundWordTokenFilter and review the attached LUCENE-8185.patch and pull request 326. Check the candidate re-check path described in the issue and add or update a regression test showing that returned terms meet minSubWordSize.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.