Make LowerCaseFilter and StopFilter keyword aware, similar to PorterStemFilter [LUCENE-3236]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
PorterStemFilter has functionality to detect if a term has been marked as a "keyword" by the KeywordMarkerFilter (KeywordAttribute.isKeyword() == true), and if so, skip stemming.
The suggestion is to have the same functionality in other filters where it is applicable. I think it may be particularly applicable to the LowerCaseFilter (ie if it is a keyword, don't mess with the case), and StopFilter (if it is a keyword, then don't filter it out even if it looks like a stop word).
Backward compatibility is maintained (in both cases) by adding a new constructor which takes an additional boolean parameter ignoreKeyword. The current constructor will call this new constructor with ignoreKeyword = false.
Patches are attached (for LowerCaseFilter and StopFilter).
I have verified that the analysis JUnit tests run against the updated code, ie, backward compatibility is maintained.
---
Migrated from [LUCENE-3236](https://issues.apache.org/jira/browse/LUCENE-3236) by Sujit Pal (@sujitpal), updated May 09 2016
Environment:
```
N/A
```
Attachments: [lucene-3236-patch.diff](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3236/lucene-3236-patch.diff), [scan.pdf](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3236/scan.pdf)
Contributor guide
Research direction
Review LowerCaseFilter and StopFilter alongside the attached lucene-3236-patch.diff, then inspect the analysis JUnit tests mentioned in the issue. Confirm how KeywordAttribute.isKeyword() should affect each filter and preserve the current constructor behavior. Done means keyword-marked terms are handled as requested while existing analysis tests and backward compatibility remain intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100