apache / apache/lucene

consider unifying stopwords formats for 5.0 [LUCENE-5227]

Open
#6,291 1 comment 0 reactions 0 assignees View on GitHub
affects-version:6.0 legacy-jira-priority:Major type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Hossman has background on #6275.

The story is we added these to lucene (it used to be a 'svn export' from snowball tree!!!!) and i had several reasons for supporting the snowball format:
1. svn export/easier to maintain diffs
2. in lucene from Analyzer APIs, as a default "Set" it didnt much matter.
3. the snowball format is nice the way they present e.g. inflection tables for some languages that inflect pronouns.

But the reality is:
1. people try to use these from e.g. solr and hit traps.
2. these things are not changing hardly at all in the snowball repository.
3. we don't do svn export anymore.
4. the "tables" could just be preserved inside # comments and still explain why the words are in the file.

We could convert our files for 5.0, and just update our stuff appropriately, and of course still support parsing the old format, and it wouldnt break anyway, just reduce traps i think.

---
Migrated from [LUCENE-5227](https://issues.apache.org/jira/browse/LUCENE-5227) by Robert Muir (@rmuir), updated May 09 2016

Contributor guide

Open the contributing guide

Research direction

Start with the background in issue #6275 and compare the current Snowball and Lucene stopword formats mentioned here. Done means converting the stopword files for 5.0, updating related Lucene usage, and retaining support for parsing the old format; the issue does not name specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.