JapaneseNumberFilter does not take whitespaces into account when concatenating numbers [LUCENE-8959]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Today the JapaneseNumberFilter tries to concatenate numbers even if they are separated by whitespaces. So for instance "10 100" is rewritten into "10100" ~~even if the tokenizer doesn't discard punctuations~~. In practice this is not an issue but this can lead to giant number of tokens if there are a lot of numbers separated by spaces. The number of concatenation should be configurable with a sane default limit in order to avoid creating giant tokens that slows down the analysis if the tokenizer is not correctly configured.
---
Migrated from [LUCENE-8959](https://issues.apache.org/jira/browse/LUCENE-8959) by Jim Ferenczi (@jimczi)
Contributor guide
Research direction
Start with JapaneseNumberFilter and reproduce the reported "10 100" behavior using numbers separated by whitespace. Determine how concatenation currently proceeds and define a configurable limit with a sensible default; done means separated numbers no longer produce unbounded giant tokens while normal concatenation remains covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100