clips / clips/pattern

Is Porter stemmer working correctly?

Open
#225 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.9k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

I wrote a Python script to check the output of pattern's implementation of the Porter2 stemmer (in the vector module) against the output of the original implementation by Martin Porter.

Martin Porter provides a test input vocabulary of 29417 words and corresponding stemmed outputs of these words obtained from his implementation of the stemmer. My script compares the output of pattern's own Porter stemmer implementation with the output of the original implementation. A total of 215 errors were found. These errors are stored in the file errors.txt by my script available here. Sample preview:

word_input original_output pattern_output
aimlessly aimless aimlessli
gazelle gazell gazel
narratives narrat narr

Pattern implements the Porter stemmer in the vector module which can be used by first importing, from pattern.vector import stem, PORTER, and then running stem(input, stemmer=PORTER). My code is available here: https://github.com/ni9elf/PatternClipsExperiments

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Porter stemmer implementation in pattern/vector and reproduce the comparison using the linked PatternClipsExperiments script and its errors.txt results. Compare the reported outputs with the original vocabulary and output files; done means determining whether the 215 differences reflect implementation bugs or expected behavior, with tests or a documented conclusion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.