apache / apache/lucene

Support for keyword protect when using ICUFoldingFilter and KeywordRepetFilter [LUCENE-9009]

Open
#10,052 1 comment 0 reactions 0 assignees View on GitHub
affects-version:7.7.2 legacy-jira-label:ICUFoldingFilterFactory legacy-jira-priority:Major module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

It would be great to support keyword protection if KeywordRepetFilter used. Like implementation at PorterStemFilter.

```java
`@Override`
public final boolean incrementToken() throws IOException {
if (!input.incrementToken())
return false;

if ((!keywordAttr.isKeyword()) && stemmer.stem(termAtt.buffer(), 0, termAtt.length()))
termAtt.copyBuffer(stemmer.getResultBuffer(), 0, stemmer.getResultLength());
return true;
}
```

 

Scenario:

We analyzing word with some accents, for example "groš". And we would like to define searching like that:

1, Search for all items which have "groš"  or "gros" in some field.

2, If we send directly search phrase "groš" we want to prefer items with word "groš" and after that with word "gros" (by the higher score). Therefore we use KeywordProtectFilter.

 

Example field definition:

```xml















```

 

The result of query and index analyzers should be like this: 

|text|groš|gros|
|-|-|-|
|raw_bytes|[67 72 6f c5 a1]|[67 72 6f c5 a1]|
|start|0|0|
|end|4|4|
|positionLength|1|1|
|type|<ALPHANUM>|<ALPHANUM>|
|termFrequency|1|1|
|position|1|1|
|keyword|true|false|

---
Migrated from [LUCENE-9009](https://issues.apache.org/jira/browse/LUCENE-9009) by Profimedia, updated Oct 18 2019

Contributor guide

Open the contributing guide

Research direction

Start with ICUFoldingFilter and KeywordRepeatFilter, then compare their token handling with the PorterStemFilter incrementToken example in the issue. Verify the analyzer behavior for “groš” and “gros”; done means protected keyword tokens remain unchanged while unprotected tokens can be folded.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.