apache / apache/lucene

WordDelimiterFilter produces invalid offsets in single word case [LUCENE-7795]

Open
#8,846 2 comments 0 reactions 0 assignees View on GitHub
affects-version:6.5 affects-version:7.0 legacy-jira-priority:Major type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

This problem is not present in WordDelimiterGraphFilter, but it is present in WordDelimiterFilter's interaction with HTMLStripCharFilter.

Test code:

```Java
public class TestTokenizationIssue2 {
public static void main(String... args) throws IOException {
HTMLStripCharFilter charFilter = new HTMLStripCharFilter(getText());
WhitespaceTokenizer whitespaceTokenizer = new WhitespaceTokenizer();
whitespaceTokenizer.setReader(charFilter);
// WordDelimiterGraphFilter wdgf = new WordDelimiterGraphFilter(whitespaceTokenizer,
// WordDelimiterGraphFilter.GENERATE_WORD_PARTS, CharArraySet.EMPTY_SET);

WordDelimiterFilter wdgf = new WordDelimiterFilter(whitespaceTokenizer,
WordDelimiterFilter.GENERATE_WORD_PARTS, CharArraySet.EMPTY_SET);
wdgf.reset();

while (wdgf.incrementToken()) {
CharTermAttribute charTermAttribute = wdgf.getAttribute(CharTermAttribute.class);
OffsetAttribute offsetAttribute = wdgf.getAttribute(OffsetAttribute.class);

System.out.println(charTermAttribute.toString() + " - " + offsetAttribute.startOffset() + ',' + offsetAttribute.endOffset());
}
}

private static Reader getText() {
return new StringReader("“Risk");
}
}
```

The offsets produced by the WordDelimiterFilter are 1,10. With WordDelimiterGraphFilter the offsets produced are 0,10. It should be 0,10 as this is the original text: ```
“Risk
``` - and 1 is between the ampersand and hash.

Inside WordDelimiterFilter, I believe the conditional branch from "if (isSingleWord && startOffset <= savedEndOffset) " is invalid and it should always use the saved start and end offsets because it can't make the assertion that the iterator's current and end are reliable markers.

---
Migrated from [LUCENE-7795](https://issues.apache.org/jira/browse/LUCENE-7795) by Michael Braun (@michaelbraun)

Contributor guide

Open the contributing guide

Research direction

Start with WordDelimiterFilter and its interaction with HTMLStripCharFilter, using the Java reproduction in the issue. Compare its offsets with WordDelimiterGraphFilter; done means the single-word case reports offsets 0,10 for the original ““Risk” text and has regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.