apache / apache/lucene

JapaneseNumberFilter does not take whitespaces into account when concatenating numbers [LUCENE-8959]

Open
#10,002 2 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Today the JapaneseNumberFilter tries to concatenate numbers even if they are separated by whitespaces. So for instance "10 100" is rewritten into "10100" ~~even if the tokenizer doesn't discard punctuations~~. In practice this is not an issue but this can lead to giant number of tokens if there are a lot of numbers separated by spaces. The number of concatenation should be configurable with a sane default limit in order to avoid creating giant tokens that slows down the analysis if the tokenizer is not correctly configured.

---
Migrated from [LUCENE-8959](https://issues.apache.org/jira/browse/LUCENE-8959) by Jim Ferenczi (@jimczi)

Contributor guide

Open the contributing guide

Research direction

Start with JapaneseNumberFilter and reproduce the reported "10 100" behavior using numbers separated by whitespace. Determine how concatenation currently proceeds and define a configurable limit with a sensible default; done means separated numbers no longer produce unbounded giant tokens while normal concatenation remains covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.