apache / apache/lucene

WordDelimiter(Graph)Filter does not handle split offsets after HTMLStripCharFilter correctly [LUCENE-7766]

Open
#8,817 0 comments 0 reactions 0 assignees View on GitHub
affects-version:6.2.1 affects-version:6.5 legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

When using the HTMLStripCharFilter before the WordDelimiterGraphFilter (or WordDelimiterFilter - I tested with both), the stripping of html from the text results in the inability to produce correct offsets for split tokens.

Configured with generate word parts, split on case change, and preserve original:

Example string: "MayBe" produces these offsets (Word - start,end)

```Java
MayBe - 0,5
May - 0,3
Be - 3,5
```

Example string "May<b>Be</b>" produces these offsets (Word- start,end)

```Java
MayBe - 0,12
May - 0,12
Be - 0,12
```

Notice that 'may' and 'be' are created but the offsets are the same as the original 'MayBe'.

I traced this down to logic within the WordDelimiterGraphFilter (and the WordDelimiterFilter before that) to how 'hasIllegalOffsets' is calculated, as is in the source code:

```Java
// if length by start + end offsets doesn't match the term's text then set offsets for all our word parts/concats to the incoming
// offsets. this can happen if WDGF is applied to an injected synonym, or to a stem'd form, etc:
hasIllegalOffsets = (savedEndOffset - savedStartOffset != savedTermLength);
```

Here is sample code that can show the issue:

```Java
public class TestTokenizationIssue {
public static void main(String... args) throws IOException {
HTMLStripCharFilter charFilter = new HTMLStripCharFilter(getText());
WhitespaceTokenizer whitespaceTokenizer = new WhitespaceTokenizer();
whitespaceTokenizer.setReader(charFilter);
WordDelimiterGraphFilter wdgf = new WordDelimiterGraphFilter(whitespaceTokenizer,
WordDelimiterGraphFilter.GENERATE_WORD_PARTS | WordDelimiterGraphFilter.SPLIT_ON_CASE_CHANGE |
WordDelimiterGraphFilter.PRESERVE_ORIGINAL, CharArraySet.EMPTY_SET);
wdgf.reset();

while (wdgf.incrementToken()) {
CharTermAttribute charTermAttribute = wdgf.getAttribute(CharTermAttribute.class);
OffsetAttribute offsetAttribute = wdgf.getAttribute(OffsetAttribute.class);

System.out.println(charTermAttribute.toString() + " - " + offsetAttribute.startOffset() + ',' + offsetAttribute.endOffset());
}
}

private static Reader getText() {
//return new StringReader("MayBe");
return new StringReader("MayBe");
}
}
```

---
Migrated from [LUCENE-7766](https://issues.apache.org/jira/browse/LUCENE-7766) by Michael Braun (@michaelbraun), 1 vote

Contributor guide

Open the contributing guide

Research direction

Start by reading the hasIllegalOffsets calculation in WordDelimiterGraphFilter and comparing it with WordDelimiterFilter, then run the Java reproduction using HTMLStripCharFilter, WhitespaceTokenizer, and the configured delimiter filter. Done means split tokens receive correct offsets after HTML stripping while preserving the expected offsets for the original and unsplit text.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.