apache / apache/lucene

Odd analysis problem with WDF, appears to be triggered by preceding analysis components [LUCENE-6689]

Open
#7,747 7 comments 0 reactions 0 assignees View on GitHub
affects-version:4.8 legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

This problem shows up for me in Solr, but I believe the issue is down at the Lucene level, so I've opened the issue in the LUCENE project. We can move it if necessary.

I've boiled the problem down to this minimum Solr fieldType:

```












```

On Solr 4.7, if this type is given the input "aaa-bbb: ccc" then index analysis puts aaa at term position 1 and bbb at term position 2. This seems perfectly reasonable to me. In Solr 4.9, both terms end up at position 2. This causes phrase queries which used to work to return zero hits. The exact text of the phrase query is in the original documents that match on 4.7.

If the custom rbbi (which is included unmodified from the lucene icu analysis source code) is not used, then the problem doesn't happen, because the punctuation doesn't make it to the PRF. If the PatternReplaceFilterFactory is not present, then the problem doesn't happen.

I can work around the problem by setting luceneMatchVersion to 4.7, but I think the behavior is a bug, and I would rather not continue to use 4.7 analysis when I upgrade to 5.x, which I hope to do soon.

Whether luceneMatchversion is LUCENE_47 or LUCENE_4_9, query analysis puts aaa at term position 1 and bbb at term position 2.

---
Migrated from [LUCENE-6689](https://issues.apache.org/jira/browse/LUCENE-6689) by Shawn Heisey (@elyograg), updated Sep 16 2015
Linked issues:
- #6175

Contributor guide

Open the contributing guide

Research direction

Start with the minimum Solr fieldType configuration and reproduce index analysis for "aaa-bbb: ccc" on the 4.7 and 4.9 behavior described. Compare term positions with and without the custom rbbi and PatternReplaceFilterFactory; done means index analysis places aaa at position 1 and bbb at position 2 while phrase queries continue to match.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.