apache / apache/lucene

Dictionary-based lemmatizer [LUCENE-6254]

Open
#7,316 6 comments 0 reactions 0 assignees View on GitHub
legacy-jira-fix-version:5.0 legacy-jira-label:patch legacy-jira-priority:Major module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The only way to achieve lemmatization today is to use the SynonymFilterFactory. The available stemmers are also inaccurate since they are only following simplistic rules.

A dictionary-based lemmatizer will be more precise because it has the opportunity to know the part of speech. Thus it provides a more precise method to stem words compared to other dictionary-based stemmers such as Hunspell.

This is my effort to develop such a lemmatizer for Apache Lucene. The documentation is temporarily placed here:
http://folk.uio.no/erlendfg/solr/lemmatizer.html

---
Migrated from [LUCENE-6254](https://issues.apache.org/jira/browse/LUCENE-6254) by Erlend Garåsen
Attachments: [LUCENE-6254.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-6254/LUCENE-6254.patch)

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked LUCENE-6254 documentation and the attached LUCENE-6254.patch to understand the proposed dictionary-based lemmatizer. Compare the proposal with Lucene's current analysis components and determine the required integration points before defining what a complete implementation and its validation would involve.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.