add sentence boundary charfilter [LUCENE-2498]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
From the discussion of #3243:
It would be nice to have a CharFilter? to mark sentence boundaries.
Such functionality would be useful for:
- prevent phrase queries with 0 slop from matching across sentences
- inhibiting multiword synonyms, or shingles, etc.
For sentence boundary detection we could use Jflex's support for the Unicode Sentence_Break property etc,
and the UAX#29 definition as a default grammar.
One idea is to just mark the boundaries with a user-provided String.
As a simple use-case, a user could then add this string to a stopfilter, and it would introduce a position increment.
This would inhibit phrase queries, etc.
a user could use the sentence-markers to do more advanced processing downstream.
---
Migrated from [LUCENE-2498](https://issues.apache.org/jira/browse/LUCENE-2498) by Robert Muir (@rmuir), updated May 16 2011
Contributor guide
Research direction
The issue names no source file, test, or entry point. Start by reading the discussion of #3243 and the migrated LUCENE-2498 issue, then review the UAX#29 Sentence_Break definition and JFlex support described here. Done should be defined by an agreed CharFilter design that marks sentence boundaries for the stated phrase-query and synonym use cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100