HTMLStripCharFilter end offset should be left of closing tags [LUCENE-5734]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Consider this simple input:
```
hello
```
to be analyzed by HTMLStripCharFilter and WhitespaceTokenizer.
You get back one token for "hello". Good. The start offset of this token is at the position of 'h' – good. But the end offset is surprisingly plus one to the adjacent </em>. I argue that it should be plus one to the last character of the token (following 'o').
FYI it behaves as I expect if after hello is an XML entity such as in this example: ```
hello
``` The end offset immediately follows the 'o'.
---
Migrated from [LUCENE-5734](https://issues.apache.org/jira/browse/LUCENE-5734) by David Smiley (@dsmiley), 2 votes, updated Jan 19 2017
Linked issues:
- #7653
Contributor guide
Research direction
Reproduce the example input with HTMLStripCharFilter and WhitespaceTokenizer, then inspect how the token end offset is calculated around the closing tag. Done means the token for "hello" ends immediately after the final character rather than at the adjacent closing tag, while the entity example remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100