apache / apache/lucene

Improved(?) Swedish snowball stemmer [LUCENE-1515]

Open
#2,589 6 comments 0 reactions 0 assignees View on GitHub
affects-version:2.4 legacy-jira-priority:Major module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Snowball stemmer for Swedish lacks support for '-an' and '-ans' related suffix stripping, ending up with non compatible stems for example "klocka", "klockor", "klockornas", "klockAN", "klockANS". Complete list of new suffix stripping rules:

{pre}
'an' 'anen' 'anens' 'anare' 'aner' 'anerna' 'anernas'
'ans' 'ansen' 'ansens' 'anser' 'ansera' 'anserar' 'anserna' 'ansernas'
'iera'
(delete)
{pre}

The problem is all the exceptions (e.g. svans|svan, finans|fin, nyans|ny) and this is an attempt at solving that problem. The rules and exceptions are based on the [SAOL](http://en.wikipedia.org/wiki/Svenska_Akademiens_Ordlista) entries suffixed with 'an' and 'ans'. There a few known problematic stemming rules but seems to work quite a bit better than the current SwedishStemmer. It would not be a bad idea to check all of SAOL entries in order to make sure the integrity of the rules.

My Snowball syntax skills are rather limited so I'm certain the code could be optimized quite a bit.

**The code is released under BSD and not ASL**. I've been posting a bit in the Snowball forum and privatly to Martin Porter himself but never got any response so now I post it here instead in hope for some momentum.

---
Migrated from [LUCENE-1515](https://issues.apache.org/jira/browse/LUCENE-1515) by Karl Wettin, updated Nov 30 2013
Attachments: [LUCENE-1515.txt](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-1515/LUCENE-1515.txt)

Contributor guide

Open the contributing guide

Research direction

Start by reading the attached LUCENE-1515.txt and comparing its proposed rules with the current SwedishStemmer. Check the listed examples and the SAOL-based exceptions, then verify that the resulting stems are compatible with the existing Swedish stemming behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.