apache / apache/lucene

TokenStream contract violation: reset()/close() call missing, [LUCENE-7227]

Open
#8,282 0 comments 0 reactions 0 assignees View on GitHub
affects-version:6.0 legacy-jira-priority:Major module:core/queryparser type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

public static void highLightClojure(int id, String text, String field) {
try {
Query queryToSearch;
TokenStream tokenStream = TokenSources.getTokenStream( field,text, analyzer);
queryToSearch = new QueryParser("asddf", analyzer).parse("read text file string utf8");
Highlighter highlighter = new Highlighter(new SimpleHTMLFormatter(),
new QueryScorer(queryToSearch));

TextFragment[] frag = highlighter.getBestTextFragments(tokenStream, text, false, 4);
for (int j = 0; j < frag.length; j++) {
if ((frag[j] != null)) {
System.out.println("score: " + frag[j].getScore() + ", frag: " + (frag[j].toString()));
}
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (InvalidTokenOffsetsException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ParseException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}

Exception:
TokenStream contract violation: reset()/close() call missing,
reset() called multiple times, or subclass does not call
super.reset(). Please see Javadocs of TokenStream class for more
information about the correct consuming workflow.

The tokenStream must be created after QueryParser, its an implicit dependency, but no any information mentioned it.

---
Migrated from [LUCENE-7227](https://issues.apache.org/jira/browse/LUCENE-7227) by Lei Zhang

Contributor guide

Open the contributing guide

Research direction

Start with the highLightClojure entry point shown in the report and reproduce the TokenStream contract exception. Trace the TokenSources, QueryParser, and Highlighter interaction to determine the expected consuming workflow; done means the reported highlighting path completes without the reset()/close() violation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.