apache / apache/lucene

Make LowerCaseFilter and StopFilter keyword aware, similar to PorterStemFilter [LUCENE-3236]

Open
#4,309 5 comments 0 reactions 0 assignees View on GitHub
affects-version:4.0-ALPHA legacy-jira-fix-version:4.9 legacy-jira-fix-version:6.0 legacy-jira-label:analysis legacy-jira-priority:Minor module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

PorterStemFilter has functionality to detect if a term has been marked as a "keyword" by the KeywordMarkerFilter (KeywordAttribute.isKeyword() == true), and if so, skip stemming.

The suggestion is to have the same functionality in other filters where it is applicable. I think it may be particularly applicable to the LowerCaseFilter (ie if it is a keyword, don't mess with the case), and StopFilter (if it is a keyword, then don't filter it out even if it looks like a stop word).

Backward compatibility is maintained (in both cases) by adding a new constructor which takes an additional boolean parameter ignoreKeyword. The current constructor will call this new constructor with ignoreKeyword = false.

Patches are attached (for LowerCaseFilter and StopFilter).

I have verified that the analysis JUnit tests run against the updated code, ie, backward compatibility is maintained.

---
Migrated from [LUCENE-3236](https://issues.apache.org/jira/browse/LUCENE-3236) by Sujit Pal (@sujitpal), updated May 09 2016
Environment:
```
N/A
```

Attachments: [lucene-3236-patch.diff](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3236/lucene-3236-patch.diff), [scan.pdf](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-3236/scan.pdf)

Contributor guide

Open the contributing guide

Research direction

Review LowerCaseFilter and StopFilter alongside the attached lucene-3236-patch.diff, then inspect the analysis JUnit tests mentioned in the issue. Confirm how KeywordAttribute.isKeyword() should affect each filter and preserve the current constructor behavior. Done means keyword-marked terms are handled as requested while existing analysis tests and backward compatibility remain intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.