apache / apache/lucene

JapanesePartOfSpeechStopFilterFactory should load built-in stop tags by default [LUCENE-9567]

Open
#10,607 4 comments 0 reactions 0 assignees View on GitHub
affects-version:8.6 legacy-jira-priority:Minor module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

If JapanesePartOfSpeechStopFilterFactory is given empty args, it does nothing. It doesn't load any stop tags, and just passes along the TokenStream passed to create().

As a default behavior, this is trappy, since a user may add the filter without explicitly adding any arguments and assume that it would load a "default" stop set. Or they may assume that if an explicit argument is required then an exception will be thrown. Regardless, "doing nothing" is almost certainly not what the user intended.

I'm going to attach a patch to load the default stop tags (using `JapaneseAnalyzer.getDefaultStopTags()`) if no args are specified, which probably makes sense in 9.0 (as it's consistent with e.g. KoreanPartOfSpeechStopFilterFactory). If we want to apply a fix to 8.x, maybe throw an exception to let the use know that the FilterFactory probably isn't doing what they think it's doing?

---
Migrated from [LUCENE-9567](https://issues.apache.org/jira/browse/LUCENE-9567) by Michael Froh (@msfroh), updated Oct 09 2020
Pull requests: https://github.com/apache/lucene-solr/pull/1961

Contributor guide

Open the contributing guide

Research direction

Start with JapanesePartOfSpeechStopFilterFactory and JapaneseAnalyzer.getDefaultStopTags(), as named in the issue. Check how empty arguments are handled and compare the expected behavior with KoreanPartOfSpeechStopFilterFactory. Done means empty arguments load the built-in stop tags or produce the intended explicit error for the relevant release.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.