Incorrect abbreviation synonyms treating in WordDelimiterFilter [LUCENE-5051]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
If there are 2 abbreviation synonyms in the stream, they are not treated as synonyms after splitting by dots in WordDelimiterFilter.
Correct treatment:
before and after WordDelimiterFilter:
```Java
tokens: wi fi wireles network
positions: 1 2 1 2
```
Incorrect treatment:
before WordDelimiterFilter:
```Java
tokens: wi.fi. wireles.network.
positions: 1 1
```
after WordDelimiterFilter:
```Java
tokens: wi fi wireles network
positions: 1 2 3 4
```
but should be:
```Java
tokens: wi wireles fi network
positions: 1 1 2 2
```
See a patch for Solr 4.3.1 configs, which demonstrates the bug if "wi.fi. router" is analyzed in name_syn field.
---
Migrated from [LUCENE-5051](https://issues.apache.org/jira/browse/LUCENE-5051) by Artem Lukanin, updated Jun 27 2013
Attachments: [incorrect_synonym_treating_sample.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5051/incorrect_synonym_treating_sample.patch), [LUCENE-5051.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5051/LUCENE-5051.patch)
Contributor guide
Research direction
Start by reviewing the attached incorrect_synonym_treating_sample.patch and LUCENE-5051.patch, then reproduce the analysis of "wi.fi. router" using the Solr 4.3.1 name_syn configuration described in the issue. Inspect WordDelimiterFilter's handling of synonym token positions and compare the output with the expected positions and ordering shown above.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100