Dictionary-based lemmatizer [LUCENE-6254]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
The only way to achieve lemmatization today is to use the SynonymFilterFactory. The available stemmers are also inaccurate since they are only following simplistic rules.
A dictionary-based lemmatizer will be more precise because it has the opportunity to know the part of speech. Thus it provides a more precise method to stem words compared to other dictionary-based stemmers such as Hunspell.
This is my effort to develop such a lemmatizer for Apache Lucene. The documentation is temporarily placed here:
http://folk.uio.no/erlendfg/solr/lemmatizer.html
---
Migrated from [LUCENE-6254](https://issues.apache.org/jira/browse/LUCENE-6254) by Erlend Garåsen
Attachments: [LUCENE-6254.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-6254/LUCENE-6254.patch)
Contributor guide
Research direction
Start by reading the linked LUCENE-6254 documentation and the attached LUCENE-6254.patch to understand the proposed dictionary-based lemmatizer. Compare the proposal with Lucene's current analysis components and determine the required integration points before defining what a complete implementation and its validation would involve.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100