apache / apache/lucene

Highlighter needs the whole text in memory to work [LUCENE-3772]

Open
#4,845 2 comments 0 reactions 0 assignees View on GitHub
affects-version:3.5 legacy-jira-label:Highlighter legacy-jira-label:improvement legacy-jira-label:memory legacy-jira-priority:Major module:highlighter type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Highlighter methods getBestFragment(s) and getBestTextFragments only accept a String object representing the whole text to highlight. When dealing with very large docs simultaneously, it can lead to heap consumption problems. It would be better if the API could accept a Reader objetct additionally, like Lucene Document Fields do.

---
Migrated from [LUCENE-3772](https://issues.apache.org/jira/browse/LUCENE-3772) by Luís Filipe Nassif, updated Oct 15 2012
Environment:
```
Windows 7 Enterprise x64, JRE 1.6.0_25
```

Contributor guide

Open the contributing guide

Research direction

Start by locating the Highlighter API methods getBestFragment(s) and getBestTextFragments and tracing how they consume the complete String. Review the related highlighting behavior and tests, then confirm that a Reader-based API can process large documents without requiring the whole text in memory while preserving existing results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.