Make CharFilter reusable [LUCENE-2788]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
The CharFilter API lets you wrap a Reader, altering the contents before the Tokenizer sees them.
It also allows you to correct the offsets so this is transparent to highlighting.
One problem is that the API isn't reusable, if you have a lot of short documents its going to be efficient.
Additionally there is some unnecessary wrapping in Tokenizer (see the CharReader.get in the ctor, but **not** in reset(Reader)!!!)
---
Migrated from [LUCENE-2788](https://issues.apache.org/jira/browse/LUCENE-2788) by Robert Muir (@rmuir), 2 votes, updated Sep 16 2012
Attachments: [LUCENE-2788.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2788/LUCENE-2788.patch)
Linked issues:
- #5300
Contributor guide
Research direction
Start by reading the CharFilter API and the Tokenizer constructor and reset(Reader) paths, especially the referenced CharReader.get behavior. Check the attached LUCENE-2788.patch and linked issue #5300 for context. Done means CharFilter can be reused for many short documents, unnecessary Tokenizer wrapping is avoided, and corrected offsets remain transparent to highlighting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100