HyphenationDecompoundTokenFilter does not set position/offset attributes correctly [LUCENE-8132]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
HyphenationDecompoundTokenFilter and DictionaryDecompoundTokenFilter set positionIncrement to 0 for all subwords, reuse start/endoffset of the original token and ignore positionLength completly.
In consequence, the QueryBuilder generates a SynonymQuery comprising all subwords, which should rather treated as individual terms.
---
Migrated from [LUCENE-8132](https://issues.apache.org/jira/browse/LUCENE-8132) by Holger Bruch, updated Jan 23 2018
Contributor guide
Research direction
Start by reading HyphenationDecompoundTokenFilter and DictionaryDecompoundTokenFilter, then trace how QueryBuilder consumes their token attributes. Reproduce the reported behavior and verify positionIncrement, start/endOffset, and positionLength for subwords. Done means subwords are treated as individual terms rather than combined into one SynonymQuery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100