apache / apache/lucene

WordDelimiterFilter stemEnglishPossessive option does not work for Tokens ending with a digit [LUCENE-5203]

Open
#6,267 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor module:analysis type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

For Token ending with "{digit}'s" the WordDelimiterFilter ignores the stemEnglishPossessive configuration

e.g. in the text "SD500's poor underwater performance" the "'s" will not be stemmed from SD500 even if stemEnglishPossessive="1".

The reason is that "WordDelimiterIterator#endsWithPossessive(int pos)" checks for

isAlpha(charType(text[pos - 3]))

while it should also allow for isDigits(int type).

---
Migrated from [LUCENE-5203](https://issues.apache.org/jira/browse/LUCENE-5203) by Rupert Westenthaler, 1 vote

Contributor guide

Open the contributing guide

Research direction

Start at WordDelimiterIterator#endsWithPossessive(int pos) and inspect how the character type before "'s" is checked. Confirm that tokens ending in a digit followed by "'s" are handled when stemEnglishPossessive is enabled, and verify the existing WordDelimiterFilter behavior for the reported SD500's example.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.