apache / apache/lucene

Set payloads in token within TokenGroup in Highlighter [LUCENE-6605]

Open
#7,663 0 comments 0 reactions 0 assignees View on GitHub
affects-version:5.2.1 legacy-jira-label:Highlighter legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I have a tokenizer which sets the payload attribute and which drives the highlighter. In Formatter.highlightTerm() I need to examine that payload on the token. However TokenGroup.addToken() doesn't currently set the payload on the token which is created and fed through to Formatter.highlightTerm() within the TokenGroup.

I would suggest, TokenGroup.addToken():

token.setPayload(payloadAtt.getPayload());

having of course registered the payload attribute during initialisation of TokenGroup with:

payloadAtt = tokenStream.addAttribute(PayloadAttribute.class);

---
Migrated from [LUCENE-6605](https://issues.apache.org/jira/browse/LUCENE-6605) by robert.onslow

Contributor guide

Open the contributing guide

Research direction

Start by locating TokenGroup.addToken() and read how TokenGroup creates the token passed to Formatter.highlightTerm(). Compare the payload attribute registration and transfer described in the issue, then verify that Formatter.highlightTerm() can observe the tokenizer's payload; done means payloads are preserved for highlighted tokens.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
Half a day
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
43/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.