JapaneseNumberFilter messes up offsets [LUCENE-10362]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
It is a tokenfilter, tries to change offsets, so of course TestRandomChains finds bugs in it
```
2> NOTE: reproduce with: gradlew test --tests TestRandomChains.testRandomChains -Dtests.seed=CE566FFD0024BDB0 -Dtests.nightly=true -Dtests.slow=true -Dtests.locale=en-PG -Dtests.timezone=CST -Dtests.asserts=true -Dtests.file.encoding=UTF-8
```
```
org.apache.lucene.analysis.tests.TestRandomChains > test suite's output saved to /home/rmuir/workspace/lucene/lucene/analysis/integration.tests/build/test-results/test_16/outputs/OUTPUT-org.apache.lucene.analysis.tests.TestRandomChains.txt, copied below:
2> stage 0: m<[0-1] +1> mi<[0-2] +1> i<[1-2] +1> iy<[1-3] +1> y<[2-3] +1> yn<[2-4] +1> n<[3-4] +1> nk<[3-5] +1> k<[4-5] +1> kt<[4-6] +1> t<[5-6] +1> t <[5-7] +1> <[6-7] +1> 2<[6-8] +1> 2<[7-8] +1> 26<[7-9] +1> 6<[8-9] +1> 64<[8-10] +1>
2> stage 1: m<[0-1] +1> mi<[0-2] +1> i<[1-2] +1> iy<[1-3] +1> y<[2-3] +1> yn<[2-4] +1> n<[3-4] +1> nk<[3-5] +1> k<[4-5] +1> kt<[4-6] +1> t<[5-6] +1> t <[5-7] +1> <[6-7] +1> 2<[6-8] +1> 2<[7-8] +1> 26<[7-9] +1> 6<[8-9] +1> 64<[8-10] +1>
2> stage 2: n<[3-4] +1> nk<[3-5] +1> word<[3-5] +0> k<[4-5] +1> word<[4-5] +0> kt<[4-6] +1> word<[4-6] +0> t<[5-6] +1> word<[5-6] +0> t <[5-7] +1> <[6-7] +1> 2<[6-8] +1> word<[6-8] +0> 2<[7-8] +1> word<[7-8] +0> 26<[7-9] +1> word<[7-9] +0> 6<[8-9] +1> 64<[8-10] +1> word<[8-10] +0>
2> last stage: yn<[2-4] +1> n<[3-4] +1> nk<[3-5] +1> word<[3-5] +0> k<[4-5] +1> word<[4-5] +0> kt<[4-6] +1> word<[4-6] +0> t<[5-6] +1> word<[5-6] +0> t <[5-7] +1> <[6-7] +1> 2<[6-8] +1> word<[6-8] +0> 2<[7-8] +1> word<[7-8] +0> 26<[7-9] +1> word<[7-9] +0> 6<[8-9] +1> word<[8-10] +0>
2> TEST FAIL: useCharFilter=false text='miynkt 264957329&#'
2> Exception from random analyzer:
2> charfilters=
2> tokenizer=
2> org.apache.lucene.analysis.ngram.NGramTokenizer()
2> filters=
2> Conditional:org.apache.lucene.analysis.icu.ICUNormalizer2Filter(OneTimeWrapper@3b5fdc7f term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1, com.ibm.icu.impl.Norm2AllModes$ComposeNormalizer2@5ef6381c)
2> Conditional:org.apache.lucene.analysis.miscellaneous.TypeAsSynonymFilter(OneTimeWrapper@3e803db2 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,flags=0, )
2> Conditional:org.apache.lucene.analysis.ja.JapaneseNumberFilter(OneTimeWrapper@20de0223 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,flags=0,keyword=false)
> java.lang.IllegalStateException: last stage: inconsistent endOffset at pos=17: 9 vs 10; token=word
> at __randomizedtesting.SeedInfo.seed([CE566FFD0024BDB0:F3B7469C4736A070]:0)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:164)
> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:1130)
```
---
Migrated from [LUCENE-10362](https://issues.apache.org/jira/browse/LUCENE-10362) by Robert Muir (@rmuir)
Contributor guide
Research direction
Start with JapaneseNumberFilter and reproduce the failure using TestRandomChains.testRandomChains with the provided Gradle command and seed. Trace the token offsets through the Japanese number filter and verify that the final token stream has consistent end offsets without the reported IllegalStateException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100