apache / apache/lucene

Incorrect abbreviation synonyms treating in WordDelimiterFilter [LUCENE-5051]

Open
#6,115 2 comments 0 reactions 0 assignees View on GitHub
affects-version:4.3 affects-version:4.3.1 legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

If there are 2 abbreviation synonyms in the stream, they are not treated as synonyms after splitting by dots in WordDelimiterFilter.

Correct treatment:
before and after WordDelimiterFilter:

```Java
tokens: wi fi wireles network
positions: 1 2 1 2
```

Incorrect treatment:
before WordDelimiterFilter:

```Java
tokens: wi.fi. wireles.network.
positions: 1 1
```

after WordDelimiterFilter:

```Java
tokens: wi fi wireles network
positions: 1 2 3 4
```

but should be:

```Java
tokens: wi wireles fi network
positions: 1 1 2 2
```

See a patch for Solr 4.3.1 configs, which demonstrates the bug if "wi.fi. router" is analyzed in name_syn field.

---
Migrated from [LUCENE-5051](https://issues.apache.org/jira/browse/LUCENE-5051) by Artem Lukanin, updated Jun 27 2013
Attachments: [incorrect_synonym_treating_sample.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5051/incorrect_synonym_treating_sample.patch), [LUCENE-5051.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5051/LUCENE-5051.patch)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the attached incorrect_synonym_treating_sample.patch and LUCENE-5051.patch, then reproduce the analysis of "wi.fi. router" using the Solr 4.3.1 name_syn configuration described in the issue. Inspect WordDelimiterFilter's handling of synonym token positions and compare the output with the expected positions and ordering shown above.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.