apache / apache/lucene

Make it easier to modify Japanese token attributes downstream [LUCENE-6216]

Open
#7,278 3 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Japanese-specific token attributes such as `PartOfSpeechAttribute`, `BaseFormAttribute`, etc. get their values from a `org.apache.lucene.analysis.ja.Token` through a `setToken()` method. This makes it cumbersome to change these token attributes later on in the analysis chain since the `Token` instances are difficult to instantiate (sort of read-only objects).

I've ran into this issue in #4995 (JapaneseNumberFilter) where it would be appropriate to update token attributes to also reflect Japanese number normalization.

I think it might be more practical to allow setting a specific value for these token attributes directly rather than through a `Token` since it makes the APIs simpler, allows for easier changing attributes downstream, and also supporting additional dictionaries easier.

The drawback with the approach that I can think of is a performance hit as we will miss out on the inherent lazy retrieval of these token attributes from the `Token` object (and the underlying dictionary/buffer).

I'd like to do some testing to better understand the performance impact of this change. Happy to hear your thoughts on this.

---
Migrated from [LUCENE-6216](https://issues.apache.org/jira/browse/LUCENE-6216) by Christian Moen (@cmoen), updated Feb 04 2015

Contributor guide

Open the contributing guide

Research direction

Start by reading the Japanese-specific token attributes, org.apache.lucene.analysis.ja.Token, and the setToken() method described in the issue. Review #4995 and JapaneseNumberFilter to understand the downstream normalization use case. Done means agreeing on an API for directly setting attribute values and measuring its performance impact against the existing Token-based approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.