apache / apache/lucene

Make CharFilter reusable [LUCENE-2788]

Open
#3,862 3 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The CharFilter API lets you wrap a Reader, altering the contents before the Tokenizer sees them.
It also allows you to correct the offsets so this is transparent to highlighting.

One problem is that the API isn't reusable, if you have a lot of short documents its going to be efficient.
Additionally there is some unnecessary wrapping in Tokenizer (see the CharReader.get in the ctor, but **not** in reset(Reader)!!!)

---
Migrated from [LUCENE-2788](https://issues.apache.org/jira/browse/LUCENE-2788) by Robert Muir (@rmuir), 2 votes, updated Sep 16 2012
Attachments: [LUCENE-2788.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2788/LUCENE-2788.patch)
Linked issues:
- #5300

Contributor guide

Open the contributing guide

Research direction

Start by reading the CharFilter API and the Tokenizer constructor and reset(Reader) paths, especially the referenced CharReader.get behavior. Check the attached LUCENE-2788.patch and linked issue #5300 for context. Done means CharFilter can be reused for many short documents, unnecessary Tokenizer wrapping is avoided, and corrected offsets remain transparent to highlighting.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.