apache / apache/lucene

FVH does not work well with phrases and multiple tags [LUCENE-7541]

Open
#8,592 1 comment 0 reactions 0 assignees View on GitHub
affects-version:trunk legacy-jira-priority:Major module:highlighter type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I'm indexing a document with a field which is `aaa bbb ccc ddd bbb eee fff`.

I'm running a Bool Query which contains 2 should Phrase queries: `aaa bbb` and `eee fff`.

I'm using an FVH with two tags `<1>` and `<2>`.

It gives the correct result: `<1>aaa bbb ccc ddd bbb <2>eee fff`

With same settings, I'm now running with 2 should Phrase queries: `aaa bbb` and `bbb eee`.

I'm getting back a wrong result: `<1>aaa bbb ccc ddd <1>bbb eee fff` where I'm expecting `<1>aaa bbb ccc ddd <2>bbb eee fff`.

Why this?

Apparently the FVH is getting back as sequence numbers in the first case `0` and `1` but in the second case `0` and `2`.

The problem is when we call then `getPreTag`, we are getting the first tag instead of the second one:

```java
protected String getPreTag( String[] preTags, int num ){
int n = num % preTags.length;
return preTags[n];
}

protected String getPostTag( String[] postTags, int num ){
int n = num % postTags.length;
return postTags[n];
}
```

I did not find yet how to fix that. But I believe it is somewhere in `org.apache.lucene.search.vectorhighlight.FieldQuery` class

```java
private void markTerminal( int slop, float boost ){
this.terminal = true;
this.slop = slop;
this.boost = boost;
this.termOrPhraseNumber = fieldQuery.nextTermOrPhraseNumber();
}
```

This call to `nextTermOrPhraseNumber()` increments the term number I guess because we have already seen the term `BBB` previously.

I'm going to join a test case patch.

---
Migrated from [LUCENE-7541](https://issues.apache.org/jira/browse/LUCENE-7541) by David Pilato (@dadoonet), updated Mar 07 2018
Attachments: [Add_test_for_FVH_with_phrase_and_multiple_tags_.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7541/Add_test_for_FVH_with_phrase_and_multiple_tags_.patch)

Contributor guide

Open the contributing guide

Research direction

Start with Add_test_for_FVH_with_phrase_and_multiple_tags_.patch and the FieldQuery.markTerminal method, then trace how nextTermOrPhraseNumber feeds getPreTag and getPostTag. Reproduce the two phrase-query cases and make the second overlapping phrase use tag 2 while preserving tag 1 for the first phrase.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.