apache / apache/lucene

JapaneseNumberFilter uses inaccurate PartOfSpeechAttribute [LUCENE-9088]

Open
#10,130 1 comment 2 reactions 0 assignees View on GitHub
legacy-jira-priority:Major module:analysis type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

According to the JapaneseNumberFilter javadocs, it uses the attribute values of the last token used to compose the normalized number, which can be wrong. While this is documented it leads to a number of incompatibilities with other japanese token filters.

For example, the PartOfSpeechAttribute of the last token used for an input text of "2008 2009" will lead to an the following output (some attributes left out...):

```

{
"token" : "2008",
"start_offset" : 0,
"end_offset" : 4,
"type" : "word",
[...]

"partOfSpeech" : "記号-空白",
"partOfSpeech (en)" : "symbol-space"

[...]
},
{
"token" : " ",
"start_offset" : 4,
"end_offset" : 5,
"type" : "word",

[...]
"partOfSpeech" : "記号-空白",
"partOfSpeech (en)" : "symbol-space",
[...]
},
{
"token" : "2009",
"start_offset" : 5,
"end_offset" : 9,
"type" : "word",
...
"partOfSpeech" : "名詞-数",
"partOfSpeech (en)" : "noun-numeric",
}

```

so that e.g. a following `kuromoji_part_of_speech` filter will eliminate the "2008" token erroneously tagged as "symbol-space".

Even without fixing the other token attrobutes, the POS attributes should IMHO be set to "noun-numeric", since that's what the filter is supposed to detect.

---
Migrated from [LUCENE-9088](https://issues.apache.org/jira/browse/LUCENE-9088) by Christoph Büscher (@cbuescher), updated Dec 11 2019

Contributor guide

Open the contributing guide

Research direction

Start by reading JapaneseNumberFilter and its javadocs, then inspect the Kuromoji part-of-speech filter and nearby tests. Reproduce the "2008 2009" example and check how normalized number tokens receive PartOfSpeechAttribute values. Done means the erroneous filtering is prevented and a regression test covers the example.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.