analysis-stempel incorrect tokens generation for numbers [LUCENE-10290]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
{**}Actual{**}:
I observed unexpected behaviour. Some numbers are affected by stemmer. It causes wrong search results.
For example "2021" -> "20ć".
{**}Expected{**}:
string numbers should not be changed.
{**}Reproduce{**}:
Issue can be reproduced with elasticsearch:
request:
```json
POST _analyze
{
"tokenizer": "standard",
"filter": ["polish_stem"],
"text": "2021"
}
```
response:
```json
{
"tokens": [
{
"token": "20ć",
"start_offset": 0,
"end_offset": 4,
"type": "",
"position": 0
}
]
}
```
I suspect the newer versions are also affected, but I don't have possibility to verify it.
---
Migrated from [LUCENE-10290](https://issues.apache.org/jira/browse/LUCENE-10290) by Dominik
Environment:
```
**Elasticsearch version** 7.11.2:
**Plugins installed**: [analysis-stempel]
**OS version** CentOS
```
Contributor guide
Research direction
Reproduce the issue with Elasticsearch's _analyze request using the standard tokenizer and the polish_stem filter, starting from the analysis-stempel component. Verify that numeric input such as "2021" remains unchanged and add a regression test covering the expected token output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100