FVH does not work well with phrases and multiple tags [LUCENE-7541]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
I'm indexing a document with a field which is `aaa bbb ccc ddd bbb eee fff`.
I'm running a Bool Query which contains 2 should Phrase queries: `aaa bbb` and `eee fff`.
I'm using an FVH with two tags `<1>` and `<2>`.
It gives the correct result: `<1>aaa bbb ccc ddd bbb <2>eee fff`
With same settings, I'm now running with 2 should Phrase queries: `aaa bbb` and `bbb eee`.
I'm getting back a wrong result: `<1>aaa bbb ccc ddd <1>bbb eee fff` where I'm expecting `<1>aaa bbb ccc ddd <2>bbb eee fff`.
Why this?
Apparently the FVH is getting back as sequence numbers in the first case `0` and `1` but in the second case `0` and `2`.
The problem is when we call then `getPreTag`, we are getting the first tag instead of the second one:
```java
protected String getPreTag( String[] preTags, int num ){
int n = num % preTags.length;
return preTags[n];
}
protected String getPostTag( String[] postTags, int num ){
int n = num % postTags.length;
return postTags[n];
}
```
I did not find yet how to fix that. But I believe it is somewhere in `org.apache.lucene.search.vectorhighlight.FieldQuery` class
```java
private void markTerminal( int slop, float boost ){
this.terminal = true;
this.slop = slop;
this.boost = boost;
this.termOrPhraseNumber = fieldQuery.nextTermOrPhraseNumber();
}
```
This call to `nextTermOrPhraseNumber()` increments the term number I guess because we have already seen the term `BBB` previously.
I'm going to join a test case patch.
---
Migrated from [LUCENE-7541](https://issues.apache.org/jira/browse/LUCENE-7541) by David Pilato (@dadoonet), updated Mar 07 2018
Attachments: [Add_test_for_FVH_with_phrase_and_multiple_tags_.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7541/Add_test_for_FVH_with_phrase_and_multiple_tags_.patch)
Contributor guide
Research direction
Start with Add_test_for_FVH_with_phrase_and_multiple_tags_.patch and the FieldQuery.markTerminal method, then trace how nextTermOrPhraseNumber feeds getPreTag and getPostTag. Reproduce the two phrase-query cases and make the second overlapping phrase use tag 2 while preserving tag 1 for the first phrase.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100