apache / apache/lucene

add sentence boundary charfilter [LUCENE-2498]

Open
#3,572 5 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

From the discussion of #3243:

It would be nice to have a CharFilter? to mark sentence boundaries.
Such functionality would be useful for:
- prevent phrase queries with 0 slop from matching across sentences
- inhibiting multiword synonyms, or shingles, etc.

For sentence boundary detection we could use Jflex's support for the Unicode Sentence_Break property etc,
and the UAX#29 definition as a default grammar.

One idea is to just mark the boundaries with a user-provided String.

As a simple use-case, a user could then add this string to a stopfilter, and it would introduce a position increment.
This would inhibit phrase queries, etc.

a user could use the sentence-markers to do more advanced processing downstream.

---
Migrated from [LUCENE-2498](https://issues.apache.org/jira/browse/LUCENE-2498) by Robert Muir (@rmuir), updated May 16 2011

Contributor guide

Open the contributing guide

Research direction

The issue names no source file, test, or entry point. Start by reading the discussion of #3243 and the migrated LUCENE-2498 issue, then review the UAX#29 Sentence_Break definition and JFlex support described here. Done should be defined by an agreed CharFilter design that marks sentence boundaries for the stated phrase-query and synonym use cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.