apache / apache/lucene

CharFilter version of ICUTransformFilter, to better support dictionary-based tokenization [LUCENE-8972]

Open
#10,015 5 comments 0 reactions 0 assignees View on GitHub
affects-version:8.2 affects-version:9.0 legacy-jira-priority:Minor module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The ICU Transliteration API is currently exposed through Lucene only post-tokinzer, via ICUTransformFilter. Some tokenizers (particularly dictionary-based) may assume pre-normalized input (e.g., for Chinese characters, there may be an assumption of traditional-only or simplified-only input characters, at the level of either all input, or per-dictionary-defined-token).

The potential usefulness of a CharFilter that exposes the ICU Transliteration API was suggested in a [thread on the Solr mailing list](https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201807.mbox/%3C4DAB7BA7-42A8-4009-8B49-60822B00DE7D%40wunderwood.org%3E), and my hope is that this issue can facilitate more detailed discussion of the proposed addition.

A concrete example of mixed traditional/simplified characters that are currently tokenized differently by the ICUTokenizer are:
- 红楼梦 (SSS)
- 紅樓夢 (TTT)
- 紅楼夢 (TST)

The first two tokens (simplified-only and traditional-only, respectively) are included in the [CJ dictionary that backs ICUTokenizer](https://raw.githubusercontent.com/unicode-org/icu/release-62-1/icu4c/source/data/brkitr/dictionaries/cjdict.txt), but the last (a mixture of traditional and simplified characters) is not, and is not recognized as a token. Even _if_ we assume this to be an intentional omission from the dictionary that results in behavior that could be desirable for some use cases, there are surely some use cases that would benefit from a more permissive dictionary-based tokenization strategy (such as could be supported by pre-tokenizer transliteration).

---
Migrated from [LUCENE-8972](https://issues.apache.org/jira/browse/LUCENE-8972) by Michael Gibney (@magibney), updated Apr 08 2022

Contributor guide

Open the contributing guide

Research direction

Start with ICUTransformFilter and ICUTokenizer, then read the linked Solr mailing-list thread and migrated LUCENE-8972. The issue needs a decision on the proposed CharFilter's scope and behavior for mixed traditional and simplified input; done is a documented design and an agreed implementation path.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.