WordDelimiterFilter produces invalid offsets in single word case [LUCENE-7795]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
This problem is not present in WordDelimiterGraphFilter, but it is present in WordDelimiterFilter's interaction with HTMLStripCharFilter.
Test code:
```Java
public class TestTokenizationIssue2 {
public static void main(String... args) throws IOException {
HTMLStripCharFilter charFilter = new HTMLStripCharFilter(getText());
WhitespaceTokenizer whitespaceTokenizer = new WhitespaceTokenizer();
whitespaceTokenizer.setReader(charFilter);
// WordDelimiterGraphFilter wdgf = new WordDelimiterGraphFilter(whitespaceTokenizer,
// WordDelimiterGraphFilter.GENERATE_WORD_PARTS, CharArraySet.EMPTY_SET);
WordDelimiterFilter wdgf = new WordDelimiterFilter(whitespaceTokenizer,
WordDelimiterFilter.GENERATE_WORD_PARTS, CharArraySet.EMPTY_SET);
wdgf.reset();
while (wdgf.incrementToken()) {
CharTermAttribute charTermAttribute = wdgf.getAttribute(CharTermAttribute.class);
OffsetAttribute offsetAttribute = wdgf.getAttribute(OffsetAttribute.class);
System.out.println(charTermAttribute.toString() + " - " + offsetAttribute.startOffset() + ',' + offsetAttribute.endOffset());
}
}
private static Reader getText() {
return new StringReader("“Risk");
}
}
```
The offsets produced by the WordDelimiterFilter are 1,10. With WordDelimiterGraphFilter the offsets produced are 0,10. It should be 0,10 as this is the original text: ```
“Risk
``` - and 1 is between the ampersand and hash.
Inside WordDelimiterFilter, I believe the conditional branch from "if (isSingleWord && startOffset <= savedEndOffset) " is invalid and it should always use the saved start and end offsets because it can't make the assertion that the iterator's current and end are reliable markers.
---
Migrated from [LUCENE-7795](https://issues.apache.org/jira/browse/LUCENE-7795) by Michael Braun (@michaelbraun)
Contributor guide
Research direction
Start with WordDelimiterFilter and its interaction with HTMLStripCharFilter, using the Java reproduction in the issue. Compare its offsets with WordDelimiterGraphFilter; done means the single-word case reports offsets 0,10 for the original ““Risk” text and has regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100