clips / clips/pattern

Should the stopword list be updated?

Open
#226 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.9k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

I wrote a Python script to compare the list of stopwords currently used by pattern's vector module against other popular stopword lists to check whether an update is required.

A total of 11 sources of stopwords were used to compare with (listed below). For each source, the set of words present in the source but not in pattern's list of stopwords and vice versa were reported in the corresponding file. Comparison output was stored in the directory Results by my code here. File x.txt contains the comparison result by comparing the stopword list x against the stopword list used by pattern.

My code is available here: https://github.com/ni9elf/PatternClipsExperiments The comparison of stopword lists is performed by the find_important and compare function of the StemmerChecker class.

Lists of stopwords compared against:

  1. unine.txt (formation retrieval multilingual resources from Universite de Neuchatel, Switzerland) link

  2. princeton.txt (Algorithms book by Robert Sedgewick and Kevin Wayne, Princeton) link

  3. nltk.txt (Natural Language Toolkit 3.2.5)

  4. yoast.txt (YoastSEO is a text analysis and assessment library in JavaScript for SEO feedback) link

  5. mysql.txt (MySQL Stopword list) link

  6. ranksnl_short.txt (Default stopword list used by ranks.nl) link

  7. ranksnl_long.txt (Longer version of stopword list used by ranks.nl) link

  8. corenlp.txt (Stanford CoreNLP - natural language software) link

  9. mallet.txt (MALLET (MAchine Learning for LanguagE Toolkit) from UMass Amherst) link

  10. glasgow.txt (Information retrieval resources from University of Glasgow) link

  11. onix.txt (Onix Text Retrieval Toolkit) link

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pattern/vector/stopwords-en.txt and review the comparison results in the linked Results directory. Read the find_important and compare functions in the StemmerChecker class, then determine whether the existing list should change and what updated contents would be justified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.