apache / apache/lucene

HTMLStripCharFilter end offset should be left of closing tags [LUCENE-5734]

Open
#6,796 13 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor module:analysis type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Consider this simple input:
```
hello
```
to be analyzed by HTMLStripCharFilter and WhitespaceTokenizer.
You get back one token for "hello". Good. The start offset of this token is at the position of 'h' – good. But the end offset is surprisingly plus one to the adjacent </em>. I argue that it should be plus one to the last character of the token (following 'o').

FYI it behaves as I expect if after hello is an XML entity such as in this example: ```
hello 
``` The end offset immediately follows the 'o'.

---
Migrated from [LUCENE-5734](https://issues.apache.org/jira/browse/LUCENE-5734) by David Smiley (@dsmiley), 2 votes, updated Jan 19 2017
Linked issues:
- #7653

Contributor guide

Open the contributing guide

Research direction

Reproduce the example input with HTMLStripCharFilter and WhitespaceTokenizer, then inspect how the token end offset is calculated around the closing tag. Done means the token for "hello" ends immediately after the final character rather than at the adjacent closing tag, while the entity example remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.