Support for keyword protect when using ICUFoldingFilter and KeywordRepetFilter [LUCENE-9009]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
It would be great to support keyword protection if KeywordRepetFilter used. Like implementation at PorterStemFilter.
```java
`@Override`
public final boolean incrementToken() throws IOException {
if (!input.incrementToken())
return false;
if ((!keywordAttr.isKeyword()) && stemmer.stem(termAtt.buffer(), 0, termAtt.length()))
termAtt.copyBuffer(stemmer.getResultBuffer(), 0, stemmer.getResultLength());
return true;
}
```
Scenario:
We analyzing word with some accents, for example "groš". And we would like to define searching like that:
1, Search for all items which have "groš" or "gros" in some field.
2, If we send directly search phrase "groš" we want to prefer items with word "groš" and after that with word "gros" (by the higher score). Therefore we use KeywordProtectFilter.
Example field definition:
```xml
```
The result of query and index analyzers should be like this:
|text|groš|gros|
|-|-|-|
|raw_bytes|[67 72 6f c5 a1]|[67 72 6f c5 a1]|
|start|0|0|
|end|4|4|
|positionLength|1|1|
|type|<ALPHANUM>|<ALPHANUM>|
|termFrequency|1|1|
|position|1|1|
|keyword|true|false|
---
Migrated from [LUCENE-9009](https://issues.apache.org/jira/browse/LUCENE-9009) by Profimedia, updated Oct 18 2019
Contributor guide
Research direction
Start with ICUFoldingFilter and KeywordRepeatFilter, then compare their token handling with the PorterStemFilter incrementToken example in the issue. Verify the analyzer behavior for “groš” and “gros”; done means protected keyword tokens remain unchanged while unprotected tokens can be folded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100