Set payloads in token within TokenGroup in Highlighter [LUCENE-6605]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
I have a tokenizer which sets the payload attribute and which drives the highlighter. In Formatter.highlightTerm() I need to examine that payload on the token. However TokenGroup.addToken() doesn't currently set the payload on the token which is created and fed through to Formatter.highlightTerm() within the TokenGroup.
I would suggest, TokenGroup.addToken():
token.setPayload(payloadAtt.getPayload());
having of course registered the payload attribute during initialisation of TokenGroup with:
payloadAtt = tokenStream.addAttribute(PayloadAttribute.class);
---
Migrated from [LUCENE-6605](https://issues.apache.org/jira/browse/LUCENE-6605) by robert.onslow
Contributor guide
Research direction
Start by locating TokenGroup.addToken() and read how TokenGroup creates the token passed to Formatter.highlightTerm(). Compare the payload attribute registration and transfer described in the issue, then verify that Formatter.highlightTerm() can observe the tokenizer's payload; done means payloads are preserved for highlighted tokens.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- Half a day
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 43/100