Improved(?) Swedish snowball stemmer [LUCENE-1515]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Snowball stemmer for Swedish lacks support for '-an' and '-ans' related suffix stripping, ending up with non compatible stems for example "klocka", "klockor", "klockornas", "klockAN", "klockANS". Complete list of new suffix stripping rules:
{pre}
'an' 'anen' 'anens' 'anare' 'aner' 'anerna' 'anernas'
'ans' 'ansen' 'ansens' 'anser' 'ansera' 'anserar' 'anserna' 'ansernas'
'iera'
(delete)
{pre}
The problem is all the exceptions (e.g. svans|svan, finans|fin, nyans|ny) and this is an attempt at solving that problem. The rules and exceptions are based on the [SAOL](http://en.wikipedia.org/wiki/Svenska_Akademiens_Ordlista) entries suffixed with 'an' and 'ans'. There a few known problematic stemming rules but seems to work quite a bit better than the current SwedishStemmer. It would not be a bad idea to check all of SAOL entries in order to make sure the integrity of the rules.
My Snowball syntax skills are rather limited so I'm certain the code could be optimized quite a bit.
**The code is released under BSD and not ASL**. I've been posting a bit in the Snowball forum and privatly to Martin Porter himself but never got any response so now I post it here instead in hope for some momentum.
---
Migrated from [LUCENE-1515](https://issues.apache.org/jira/browse/LUCENE-1515) by Karl Wettin, updated Nov 30 2013
Attachments: [LUCENE-1515.txt](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-1515/LUCENE-1515.txt)
Contributor guide
Research direction
Start by reading the attached LUCENE-1515.txt and comparing its proposed rules with the current SwedishStemmer. Check the listed examples and the SAOL-based exceptions, then verify that the resulting stems are compatible with the existing Swedish stemming behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100